Strength of Minibatch Noise in SGD

2021/02/10 by Liu Ziyin, Kangqiao Liu, Ziyin, Liu +6 · 6 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Neural Networks and Applications #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2102.05375

ICLR 2022 spotlight

openalex publication_date 2021/02/10 · arxiv created 2022/03/08 · arxiv updated 2022/03/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning. This work presents the first systematic study of the SGD noise and fluctuations close to a local minimum. We first analyze the SGD noise in linear regression in detail and then derive a general formula for approximating SGD noise in different types of minima. For application, our results (1) provide insight into the stability of training a neural network, (2) suggest that a large learning rate can help generalization by introducing an implicit regularization, (3) explain why the linear learning rate-batchsize scaling law fails at a large learning rate or at a small batchsize and (4) can provide an understanding of how discrete-time nature of SGD affects the recently discovered power-law phenomenon of SGD.

Citations

• Natural Gradient Works Efficiently in Learning
• Computational Methods for Linear Matrix Equations
• A Bayesian Perspective on Generalization and Stochastic Gradient Descent
• Understanding the Role of Momentum in Stochastic Gradient Methods
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance

Cited by

• Training Across Reservoirs: Using Numerical Differentiation To Couple Trainable Networks With Black-Box Reservoirs
• Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise
• FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learning
• Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent
• SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training
• Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning

Related

• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• On the Noisy Gradient Descent that Generalizes as SGD
• Improved generalization by noise enhancement
• Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning
• SGD with a Constant Large Learning Rate Can Converge to Local Maxima