The Effect of Network Width on the Performance of Large-batch Training

2018/06/11 by Lingjiao Chen, Hongyi Wang, Chen, Lingjiao +7
Computer Science · Mathematics · #Computation (stat.CO) #Distributed #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Parallel #and Cluster Computing (cs.DC) #cs.DC #cs.LG #math.OC #stat.CO #stat.ML

paper · pdf · doi:10.48550/arxiv.1806.03791

arxiv created 2018/06/11 · arxiv updated 2018/06/12

Abstract

Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and generalization performance of SGD. In this work, we take a first step towards analyzing how the structure (width and depth) of a neural network affects the performance of large-batch training. We present new theoretical results which suggest that--for a fixed number of parameters--wider networks are more amenable to fast large-batch training compared to deeper ones. We provide extensive experiments on residual and fully-connected neural networks which suggest that wider networks can be trained using larger batches without incurring a convergence slow-down, unlike their deeper variants.

Related

• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Don't Use Large Mini-Batches, Use Local SGD
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Special Properties of Gradient Descent with Large Learning Rates