Label Noise SGD Provably Prefers Flat Global Minimizers

2021/06/11 by Alex Damian, Tengyu Ma, Damian, Alex +3 · 10 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Mathematics #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2106.06530

openalex publication_date 2021/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In overparametrized models, the noise in stochastic gradient descent (SGD) implicitly regularizes the optimization trajectory and determines which local minimum SGD converges to. Motivated by empirical studies that demonstrate that training with noisy labels improves generalization, we study the implicit regularization effect of SGD with label noise. We show that SGD with label noise converges to a stationary point of a regularized loss L(θ) +λR(θ), where L(θ) is the training loss, λ is an effective regularization parameter depending on the step size, strength of the label noise, and the batch size, and R(θ) is an explicit regularizer that penalizes sharp minimizers. Our analysis uncovers an additional regularization effect of large learning rates beyond the linear scaling rule that penalizes large eigenvalues of the Hessian more than small ones. We also prove extensions to classification with general loss functions, SGD with momentum, and SGD with general noise covariance, significantly strengthening the prior work of Blanc et al. to global convergence and large learning rates and of HaoChen et al. to general models.

Citations

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Learning Multiple Layers of Features from Tiny Images
• Gradient Descent on Neural Networks Typically Occurs at the Edge of\n Stability

Cited by

• Flat Minima and Generalization: Insights from Stochastic Convex Optimization
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• A Stochastic Algorithm for Searching Saddle Points with Convergence Guarantee
• Flatness-Aware Stochastic Gradient Langevin Dynamics
• Rényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization
• ATOM: A Pretrained Neural Operator for Multitask Molecular Dynamics
• Indirect Attention: Turning Context Misalignment into a Feature
• Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise
• Optimal Gradient-based Algorithms for Non-concave Bandit Optimization
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Related

• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• The alignment property of SGD noise and how it helps select flat minima: A stability analysis
• Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability
• Bad Global Minima Exist and SGD Can Reach Them