2017/12/08 by Shankar Krishnan, Ying Xiao, Krishnan, Shankar +3
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.1712.03298
openalex publication_date 2017/12/08 · openalex created_date 2022/09/30 · openalex updated_date 2026/07/28
Progress in deep learning is slowed by the days or weeks it takes to train\nlarge models. The natural solution of using more hardware is limited by\ndiminishing returns, and leads to inefficient use of additional resources. In\nthis paper, we present a large batch, stochastic optimization algorithm that is\nboth faster than widely used algorithms for fixed amounts of computation, and\nalso scales up substantially better as more computational resources become\navailable. Our algorithm implicitly computes the inverse Hessian of each\nmini-batch to produce descent directions; we do so without either an explicit\napproximation to the Hessian or Hessian-vector products. We demonstrate the\neffectiveness of our algorithm by successfully training large ImageNet models\n(Inception-V3, Resnet-50, Resnet-101 and Inception-Resnet-V2) with mini-batch\nsizes of up to 32000 with no loss in validation error relative to current\nbaselines, and no increase in the total number of steps. At smaller mini-batch\nsizes, our optimizer improves the validation error in these models by 0.8-0.9%.\nAlternatively, we can trade off this accuracy to reduce the number of training\nsteps needed by roughly 10-30%. Our work is practical and easily usable by\nothers -- only one hyperparameter (learning rate) needs tuning, and\nfurthermore, the algorithm is as computationally cheap as the commonly used\nAdam optimizer.\n