Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash

2018/11/13 by Hiroaki Mikami, Mikami, Hiroaki, Hisahiro Suganuma +8 · 10 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Stochastic Gradient Optimization Techniques #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.1811.05233

openalex publication_date 2018/11/13 · arxiv created 2019/03/05 · arxiv updated 2019/03/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Scaling the distributed deep learning to a massive GPU cluster level is challenging due to the instability of the large mini-batch training and the overhead of the gradient synchronization. We address the instability of the large mini-batch training with batch-size control and label smoothing. We address the overhead of the gradient synchronization with 2D-Torus all-reduce. Specifically, 2D-Torus all-reduce arranges GPUs in a logical 2D grid and performs a series of collective operation in different orientations. These two techniques are implemented with Neural Network Libraries (NNL). We have successfully trained ImageNet/ResNet-50 in 122 seconds without significant accuracy loss on ABCI cluster.

Citations

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Large batch size training of neural networks with adversarial training and second-order information

Cited by

• Bandwidth-Aware Network Topology Optimization for Decentralized Learning
• Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
• Optimization for deep learning: theory and algorithms
• A Generalization of the Allreduce Operation
• Distributed Deep Learning in Open Collaborations
• Augment your batch: better training with larger batches
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware

Related

• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• How to scale distributed deep learning?
• A Hitchhiker's Guide On Distributed Training of Deep Neural Networks