Exoshuffle-CloudSort

2023/01/10 by Luan, Frank Sifei, Wang, Stephanie, Yagati, Samyukta +7
#Distributed #FOS: Computer and information sciences #Operating Systems (cs.OS) #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2301.03734

Abstract

We present Exoshuffle-CloudSort, a sorting application running on top of Ray using the Exoshuffle architecture. Exoshuffle-CloudSort runs on Amazon EC2, with input and output data stored on Amazon S3. Using 40 i4i.4xlarge workers, Exoshuffle-CloudSort completes the 100 TB CloudSort Benchmark (Indy category) in 5378 seconds, with an average total cost of 97.

Related

• Syndeo: Portable Ray Clusters with Secure Containerization
• Thrill: High-Performance Algorithmic Distributed Batch Data Processing with C++
• Exosphere -- Bringing The Cloud Closer
• SProBench: Stream Processing Benchmark for High Performance Computing Infrastructure
• XaaS: Acceleration as a Service to Enable Productive High-Performance Cloud Computing