Extrapolation for Large-batch Training in Deep Learning

2020/06/10 by Tao Lin, Lin, Tao, Lingjing Kong +5 · 3 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Computer science #Convergence (economics) #Deep learning #Domain Adaptation and Few-Shot Learning #Extrapolation #Fraction (chemistry) #Generalization #Gradient descent #Machine learning #Mathematical optimization #Mathematics #Maxima and minima #Scaling #Scheme (mathematics) #Smoothing #Stochastic gradient descent #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.05720

published in arXiv (Cornell University) 1, 6094-6104 (Cornell University)

arxiv created 2020/06/10 · openalex publication_date 2020/06/10 · arxiv updated 2020/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batch size to a substantial fraction of the training data for improving training time is the persistent degradation in performance (generalization gap). To address this issue, recent work propose to add small perturbations to the model parameters when computing the stochastic gradients and report improved generalization performance due to smoothing effects. However, this approach is poorly understood; it requires often model-specific noise and fine-tuning. To alleviate these drawbacks, we propose to use instead computationally efficient extrapolation (extragradient) to stabilize the optimization trajectory while still benefiting from smoothing to avoid sharp minima. This principled approach is well grounded from an optimization perspective and we show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer. We demonstrate that in a variety of experiments the scheme allows scaling to much larger batch sizes than before whilst reaching or surpassing SOTA accuracy.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Densely Connected Convolutional Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Pointer Sentinel Mixture Models
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Cited by

• On Using Large-Batches in Federated Learning
• MuLoCo: Muon is a practical inner optimizer for DiLoCo
• Convergence of First-Order Algorithms with Momentum from the Perspective of an Inexact Gradient Descent Method

Related

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Augment your batch: better training with larger batches
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Don't Use Large Mini-Batches, Use Local SGD