Efficient Training of Convolutional Neural Nets on Large Distributed\n Systems

2017/11/02 by Sameer Kumar, Kumar, Sameer, Dheeraj Sreedhar +7
Computer Science · Neuroscience · #Advanced Neural Network Applications #Brain Tumor Detection and Classification #Distributed #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.1711.00705

openalex publication_date 2017/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep Neural Networks (DNNs) have achieved im- pressive accuracy in many\napplication domains including im- age classification. Training of DNNs is an\nextremely compute- intensive process and is solved using variants of the\nstochastic gradient descent (SGD) algorithm. A lot of recent research has\nfocussed on improving the performance of DNN training. In this paper, we\npresent optimization techniques to improve the performance of the data parallel\nsynchronous SGD algorithm using the Torch framework: (i) we maintain data\nin-memory to avoid file I/O overheads, (ii) we present a multi-color based MPI\nAllreduce algorithm to minimize communication overheads, and (iii) we propose\noptimizations to the Torch data parallel table framework that handles\nmulti-threading. We evaluate the performance of our optimizations on a Power 8\nMinsky cluster with 32 nodes and 128 NVidia Pascal P100 GPUs. With our\noptimizations, we are able to train 90 epochs of the ResNet-50 model on the\nImagenet-1k dataset using 256 GPUs in just 48 minutes. This significantly\nimproves on the previously best known performance of training 90 epochs of the\nResNet-50 model on the same dataset using 256 GPUs in 65 minutes. To the best\nof our knowledge, this is the best known training performance demonstrated for\nthe Imagenet- 1k dataset.\n

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Residual Learning for Image Recognition

Related

• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Optimizing Distributed Training Approaches for Scaling Neural Networks
• HyperTune: Dynamic Hyperparameter Tuning For Efficient Distribution of DNN Training Over Heterogeneous Systems