2023/11/06 by Kabir Nagrecha, Arun Kumar, Nagrecha, Kabir +1
Computer Science · #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2311.02840
openalex publication_date 2023/11/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this paper, we propose Saturn, a new data system to improve the efficiency of multi-large-model training (e.g., during model selection/hyperparameter optimization). We first identify three key interconnected systems challenges for users building large models in this setting -- parallelism technique selection, distribution of GPUs over jobs, and scheduling. We then formalize these as a joint problem, and build a new system architecture to tackle these challenges simultaneously. Our evaluations show that our joint-optimization approach yields 39-49% lower model selection runtimes than typical current DL practice.