2020/02/20 by André Luckow, Luckow, Andre, Shantenu Jha +1
Computer Science · Decision Sciences · #Cloud Computing and Resource Management #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Parallel #Scientific Computing and Data Management #Software Engineering (cs.SE) #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2002.09009
openalex publication_date 2020/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Developing software for scientific applications that require the integration\nof diverse types of computing, instruments, and data present challenges that\nare distinct from commercial software. These applications require scale, and\nthe need to integrate various programming and computational models with\nevolving and heterogeneous infrastructure. Pervasive and effective abstractions\nfor distributed infrastructures are thus critical; however, the process of\ndeveloping abstractions for scientific applications and infrastructures is not\nwell understood. While theory-based approaches for system development are\nsuited for well-defined, closed environments, they have severe limitations for\ndesigning abstractions for scientific systems and applications. The design\nscience research (DSR) method provides the basis for designing practical\nsystems that can handle real-world complexities at all levels. In contrast to\ntheory-centric approaches, DSR emphasizes both practical relevance and\nknowledge creation by building and rigorously evaluating all artifacts. We show\nhow DSR provides a well-defined framework for developing abstractions and\nmiddleware systems for distributed systems. Specifically, we address the\ncritical problem of distributed resource management on heterogeneous\ninfrastructure over a dynamic range of scales, a challenge that currently\nlimits many scientific applications. We use the pilot-abstraction, a widely\nused resource management abstraction for high-performance, high throughput, big\ndata, and streaming applications, as a case study for evaluating the DSR\nactivities. For this purpose, we analyze the research process and artifacts\nproduced during the design and evaluation of the pilot-abstraction. We find DSR\nprovides a concise framework for iteratively designing and evaluating systems.\nFinally, we capture our experiences and formulate different lessons learned.\n