On the Generalization Benefit of Noise in Stochastic Gradient Descent

2020/06/26 by Samuel Smith, Erich Elsen, Smith, Samuel L. +3 · 8 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2006.15081

openalex publication_date 2020/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

It has long been argued that minibatch stochastic gradient descent can generalize better than large batch gradient descent in deep neural networks. However recent papers have questioned this claim, arguing that this effect is simply a consequence of suboptimal hyperparameter tuning or insufficient compute budgets when the batch size is large. In this paper, we perform carefully designed experiments and rigorous hyperparameter sweeps on a range of popular models, which verify that small or moderately large batch sizes can substantially outperform very large batches on the test set. This occurs even when both models are trained for the same number of iterations and large batches achieve smaller training losses. Our results confirm that the noise in stochastic gradients can enhance generalization. We study how the optimal learning rate schedule changes as the epoch budget grows, and we provide a theoretical account of our observations based on the stochastic differential equation perspective of SGD dynamics.

Cited by

• Scaling Behavior of Discrete Diffusion Language Models
• Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks
• Improving Quantum Neural Networks exploration by Noise-Induced Equalization
• Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules
• How does the optimizer implicitly bias the model merging loss landscape?
• Variational Learning Finds Flatter Solutions at the Edge of Stability
• On the Interaction of Noise, Compression Role, and Adaptivity under (L₀, L₁)-Smoothness: An SDE-based Approach
• SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training

Related

• Special Properties of Gradient Descent with Large Learning Rates
• A Bayesian Perspective on Generalization and Stochastic Gradient Descent
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Strength of Minibatch Noise in SGD
• Three Factors Influencing Minima in SGD