Improved generalization by noise enhancement

2020/09/28 by T. Mori, Takashi Mori, Masahito Ueda +2
Computer Science · Engineering · Mathematics · Physics and Astronomy · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Sparse and Compressive Sensing Techniques #cond-mat.dis-nn #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2009.13094

9 pages

arxiv created 2020/09/28 · openalex publication_date 2020/09/28 · arxiv updated 2020/09/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent studies have demonstrated that noise in stochastic gradient descent (SGD) is closely related to generalization: A larger SGD noise, if not too large, results in better generalization. Since the covariance of the SGD noise is proportional to η²/B, where η is the learning rate and B is the minibatch size of SGD, the SGD noise has so far been controlled by changing η and/or B. However, too large η results in instability in the training dynamics and a small B prevents scalable parallel computation. It is thus desirable to develop a method of controlling the SGD noise without changing η and B. In this paper, we propose a method that achieves this goal using ``noise enhancement'', which is easily implemented in practice. We expound the underlying theoretical idea and demonstrate that the noise enhancement actually improves generalization for real datasets. It turns out that large-batch training with the noise enhancement even shows better generalization compared with small-batch training.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Three Factors Influencing Minima in SGD
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• On Exact Computation with an Infinitely Wide Neural Net
• Non-Gaussianity of Stochastic Gradient Noise
• On the Noisy Gradient Descent that Generalizes as SGD

Related

• On the Noisy Gradient Descent that Generalizes as SGD
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Strength of Minibatch Noise in SGD
• Anticorrelated Noise Injection for Improved Generalization
• Noisy Truncated SGD: Optimization and Generalization