SGD Implicitly Regularizes Generalization Error

2021/04/10 by Daniel A. Roberts, Roberts, Daniel A. · 3 citations
Computer Science · Medicine · #Advanced Neuroimaging Techniques and Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2104.04874

openalex publication_date 2021/04/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochastic gradient descent to the change in test error from an equivalent number of gradient descent updates and show explicitly that stochastic gradient descent acts to regularize generalization error by decorrelating nearby updates. These calculations depends on the details of the model only through the mean and covariance of the gradient distribution, which may be readily measured for particular models of interest. We discuss further improvements to these calculations and comment on possible implications for stochastic optimization.

Cited by

• Generalization by design: Shortcuts to Generalization in Deep Learning
• Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning
• Overfitting has a limitation: a model-independent generalization gap bound based on Rényi entropy

Related

• Disentangling the Mechanisms Behind Implicit Regularization in SGD
• Label Noise SGD Provably Prefers Flat Global Minimizers
• Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
• Generalization Properties and Implicit Regularization for Multiple Passes SGM
• Improved generalization by noise enhancement