2019/09/13 by André Luckow, Luckow, Andre, Shantenu Jha +1
Computer Science · Decision Sciences · #Cloud Computing and Resource Management #Distributed #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Parallel #Scientific Computing and Data Management #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.1909.06055
openalex publication_date 2019/09/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Experiment-in-the-Loop Computing (EILC) requires support for numerous types\nof processing and the management of heterogeneous infrastructure over a dynamic\nrange of scales: from the edge to the cloud and HPC, and intermediate\nresources. Serverless is an emerging service that combines high-level\nmiddleware services, such as distributed execution engines for managing tasks,\nwith low-level infrastructure. It offers the potential of usability and\nscalability, but adds to the complexity of managing heterogeneous and dynamic\nresources. In response, we extend Pilot-Streaming to support serverless\nplatforms. Pilot-Streaming provides a unified abstraction for resource\nmanagement for HPC, cloud, and serverless, and allocates resource containers\nindependent of the application workload removing the need to write\nresource-specific code. Understanding of the performance and scaling\ncharacteristics of streaming applications and infrastructure presents another\nchallenge for EILC. StreamInsight provides insight into the performance of\nstreaming applications and infrastructure, their selection, configuration and\nscaling behavior. Underlying StreamInsight is the universal scalability law,\nwhich permits the accurate quantification of scalability properties of\nstreaming applications. Using experiments on HPC and AWS Lambda, we demonstrate\nthat StreamInsight provides an accurate model for a variety of application\ncharacteristics, e.g., machine learning model sizes and resource\nconfigurations.\n