2018/01/25 by André Luckow, Luckow, Andre, George Chantzialexiou +3
Computer Science · Decision Sciences · #Advanced Data Storage Technologies #Cloud Computing and Resource Management #Distributed #FOS: Computer and information sciences #Parallel #Scientific Computing and Data Management #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.1801.08648
openalex publication_date 2018/01/25 · openalex created_date 2022/09/26 · openalex updated_date 2026/07/28
An increasing number of scientific applications rely on stream processing for\ngenerating timely insights from data feeds of scientific instruments,\nsimulations, and Internet-of-Thing (IoT) sensors. The development of streaming\napplications is a complex task and requires the integration of heterogeneous,\ndistributed infrastructure, frameworks, middleware and application components.\nDifferent application components are often written in different languages using\ndifferent abstractions and frameworks. Often, additional components, such as a\nmessage broker (e.g. Kafka), are required to decouple data production and\nconsumptions and avoiding issues, such as back-pressure. Streaming applications\nmay be extremely dynamic due to factors, such as variable data rates caused by\nthe data source, adaptive sampling techniques or network congestions, variable\nprocessing loads caused by usage of different machine learning algorithms. As a\nresult application-level resource management that can respond to changes in one\nof these factors is critical. We propose Pilot-Streaming, a framework for\nsupporting streaming frameworks, applications and their resource management\nneeds on HPC infrastructure. Pilot-Streaming is based on the Pilot-Job concept\nand enables developers to manage distributed computing and data resources for\ncomplex streaming applications. It enables applications to dynamically respond\nto resource requirements by adding/removing resources at runtime. This\ncapability is critical for balancing complex streaming pipelines. To address\nthe complexity in developing and characterization of streaming applications, we\npresent the Streaming Mini- App framework, which supports different plug-able\nalgorithms for data generation and processing, e.g., for reconstructing light\nsource images using different techniques. We utilize the Mini-App framework to\nconduct an evaluation of Pilot-Streaming.\n