Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics

2021/09/20 by Yixin Wu, Wu, Yixin, Rui Luo +7
Computer Science · Mathematics · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Markov Chains and Monte Carlo Methods #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2109.09833

18 pages

arxiv created 2021/09/20 · openalex publication_date 2021/09/20 · arxiv updated 2021/09/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we characterize the noise of stochastic gradients and analyze the noise-induced dynamics during training deep neural networks by gradient-based optimizers. Specifically, we firstly show that the stochastic gradient noise possesses finite variance, and therefore the classical Central Limit Theorem (CLT) applies; this indicates that the gradient noise is asymptotically Gaussian. Such an asymptotic result validates the wide-accepted assumption of Gaussian noise. We clarify that the recently observed phenomenon of heavy tails within gradient noise may not be intrinsic properties, but the consequence of insufficient mini-batch size; the gradient noise, which is a sum of limited i.i.d. random variables, has not reached the asymptotic regime of CLT, thus deviates from Gaussian. We quantitatively measure the goodness of Gaussian approximation of the noise, which supports our conclusion. Secondly, we analyze the noise-induced dynamics of stochastic gradient descent using the Langevin equation, granting for momentum hyperparameter in the optimizer with a physical interpretation. We then proceed to demonstrate the existence of the steady-state distribution of stochastic gradient descent and approximate the distribution at a small learning rate.

Citations

• An Extension of Shapiro and Wilk's W Test for Normality to Large Samples
• The Principles of Quantum Mechanics
• A Stochastic Approximation Method
• ImageNet classification with deep convolutional neural networks
• An analysis of variance test for normality (complete samples)
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Thermostat-assisted continuously-tempered Hamiltonian Monte Carlo for Bayesian learning
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Non-Gaussianity of Stochastic Gradient Noise
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Game-Theoretic Multiagent Reinforcement Learning
• Highly accurate protein structure prediction with AlphaFold

Related

• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Revisiting the Noise Model of Stochastic Gradient Descent
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• On the Overlooked Structure of Stochastic Gradients
• Non-Gaussianity of Stochastic Gradient Noise