Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

2017/11/12 by Akiba, Takuya, Suzuki, Shuji, Fukuda, Keisuke
#Computer Vision and Pattern Recognition (cs.CV) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.1711.04325

Abstract

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.

Related

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• ImageNet Training in Minutes
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash