Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash

2018/11/13 by Mikami, Hiroaki, Suganuma, Hisahiro, U-chupala, Pongsakorn +2 · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1811.05233

Abstract

Scaling the distributed deep learning to a massive GPU cluster level is challenging due to the instability of the large mini-batch training and the overhead of the gradient synchronization. We address the instability of the large mini-batch training with batch-size control and label smoothing. We address the overhead of the gradient synchronization with 2D-Torus all-reduce. Specifically, 2D-Torus all-reduce arranges GPUs in a logical 2D grid and performs a series of collective operation in different orientations. These two techniques are implemented with Neural Network Libraries (NNL). We have successfully trained ImageNet/ResNet-50 in 122 seconds without significant accuracy loss on ABCI cluster.

Cited by

• Bandwidth-Aware Network Topology Optimization for Decentralized Learning

Related

• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• How to scale distributed deep learning?
• A Hitchhiker's Guide On Distributed Training of Deep Neural Networks