Non-Gaussianity of Stochastic Gradient Noise

2019/10/21 by Abhishek Panigrahi, Panigrahi, Abhishek, Raghav Somani +5 · 23 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Applied mathematics #Artificial intelligence #Artificial neural network #Computer science #Distribution (mathematics) #FOS: Computer and information sciences #Gaussian #Gaussian Processes and Bayesian Inference #Gaussian noise #Generalization #Gradient descent #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical analysis #Mathematics #Noise (video) #Physics #Statistical physics #Stochastic Gradient Optimization Techniques #Stochastic gradient descent #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1910.09626

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2019/10/21 · arxiv created 2019/10/25 · arxiv updated 2019/10/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes 256 and above, the distribution is best described as Gaussian at-least in the early phases of training. This holds across data-sets, architectures, and other choices.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Stochastic Approximation Method
• ImageNet classification with deep convolutional neural networks
• An analysis of variance test for normality (complete samples)
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• Three Factors Influencing Minima in SGD
• On the diffusion approximation of nonconvex stochastic gradient descent
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Quantifying Information Disclosure During Gradient Descent Using Gradient Uniqueness
• On the Noisy Gradient Descent that Generalizes as SGD
• Variational Learning Finds Flatter Solutions at the Edge of Stability
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• Fractional moment-preserving initialization schemes for training deep neural networks
• Improved generalization by noise enhancement
• An Empirical Study on the Intrinsic Privacy of SGD

Related

• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• Revisiting the Noise Model of Stochastic Gradient Descent
• Special Properties of Gradient Descent with Large Learning Rates
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks