2020/01/01 by Vinu E. Venugopal, Venugopal, Vinu E., Martin Theobald +5 · 1 voice
Computer Science · Decision Sciences · Engineering · #Advanced Database Systems and Queries #Asynchronous communication #Cloud Computing and Resource Management #Computer network #Computer science #Dataflow #Distributed #Distributed computing #FOS: Computer and information sciences #FOS: Electrical engineering #InfiniBand #Message queue #Parallel #Parallel computing #SPARK (programming language) #Scientific Computing and Data Management #Stream processing #Systems and Control (eess.SY) #and Cluster Computing (cs.DC) #cs.DC #cs.SY #eess.SY #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2001.00164
published in arXiv (Cornell University) (Cornell University) · 16 pages, 6 figures, 15 plots
arxiv published 2020/01/01 · arxiv created 2020/01/03 · arxiv updated 2020/01/06
Distributed Stream Processing Systems (DSPSs) are among the currently most\nemerging topics in data management, with applications ranging from real-time\nevent monitoring to processing complex dataflow programs and big data\nanalytics. The major market players in this domain are clearly represented by\nApache Spark and Flink, which provide a variety of frontend APIs for SQL,\nstatistical inference, machine learning, stream processing, and many others.\nYet rather few details are reported on the integration of these engines into\nthe underlying High-Performance Computing (HPC) infrastructure and the\ncommunication protocols they use. Spark and Flink, for example, are implemented\nin Java and still rely on a dedicated master node for managing their control\nflow among the worker nodes in a compute cluster.\n In this paper, we describe the architecture of our AIR engine, which is\ndesigned from scratch in C++ using the Message Passing Interface (MPI),\npthreads for multithreading, and is directly deployed on top of a common HPC\nworkload manager such as SLURM. AIR implements a light-weight, dynamic sharding\nprotocol (referred to as "Asynchronous Iterative Routing"), which facilitates a\ndirect and asynchronous communication among all client nodes and thereby\ncompletely avoids the overhead induced by the control flow with a master node\nthat may otherwise form a performance bottleneck. Our experiments over a\nvariety of benchmark settings confirm that AIR outperforms Spark and Flink in\nterms of latency and throughput by a factor of up to 15; moreover, we\ndemonstrate that AIR scales out much better than existing DSPSs to clusters\nconsisting of up to 8 nodes and 224 cores.\n