2023/02/22 by Zhuohan Li, Lianmin Zheng, Li, Zhuohan +19 · 8 citations
Computer Science · #Age of Information Optimization #Cloud Computing and Resource Management #Distributed #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Machine Learning (cs.LG) #Networking and Internet Architecture (cs.NI) #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2302.11665
openalex publication_date 2023/02/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests.