On Scale-out Deep Learning Training for Cloud and HPC

2018/01/24 by Srinivas Sridharan, Sridharan, Srinivas, Karthikeyan Vaidyanathan +19 · 3 citations
Computer Science · #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Parallel Computing and Optimization Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #cs.DC #cs.LG

paper · pdf · doi:10.48550/arxiv.1801.08030

Accepted in SysML 2018 conference

arxiv created 2018/01/24 · openalex publication_date 2018/01/24 · arxiv updated 2018/01/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes. This can only be achieved through scalable and efficient distributed training, since a single node/card cannot satisfy the compute, memory, and I/O requirements of today's state-of-the-art deep neural networks. However, scaling synchronous Stochastic Gradient Descent (SGD) is still a challenging problem and requires continued research/development. This entails innovations spanning algorithms, frameworks, communication libraries, and system design. In this paper, we describe the philosophy, design, and implementation of Intel Machine Learning Scalability Library (MLSL) and present proof-points demonstrating scaling DL training on 100s to 1000s of nodes across Cloud and HPC systems.

Citations

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Don't Decay the Learning Rate, Increase the Batch Size

Cited by

• Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning
• High-Performance Deep Learning via a Single Building Block
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems

Related

• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
• Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems
• Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes