Fluctuation-dissipation relations for stochastic gradient descent

2018/09/28 by Sho Yaida, Yaida, Sho · 23 citations
Computer Science · Mathematics · #Applied mathematics #Artificial intelligence #Artificial neural network #Computer science #Descent (aeronautics) #FOS: Computer and information sciences #Function (biology) #Gaussian Processes and Bayesian Inference #Hessian matrix #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Mathematics #Physics #Set (abstract data type) #Stationary state #Statistical physics #Stochastic Gradient Optimization Techniques #Stochastic gradient descent #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1810.00004

published in arXiv (Cornell University) (Cornell University) · 15 pages, 6 figures; v2: final version accepted at ICLR 2019, with derivations/assumptions clarified and Adam/AMSGrad experiments added

openalex publication_date 2018/09/28 · arxiv created 2018/12/21 · arxiv updated 2018/12/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm. These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule. We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity. Our claims are empirically verified.

Citations

• A Stochastic Approximation Method
• Statistical-Mechanical Theory of Irreversible Processes. I. General Theory and Simple Applications to Magnetic and Conduction Problems
• Über die von der molekularkinetischen Theorie der Wärme geforderte Bewegung von in ruhenden Flüssigkeiten suspendierten Teilchen
• Gradient-based learning applied to document recognition
• Three Factors Influencing Minima in SGD
• Learning Multiple Layers of Features from Tiny Images

Cited by

• The Universe Learning Itself: On the Evolution of Dynamics from the Big Bang to Machine Intelligence
• Contribution of task-irrelevant stimuli to drift of neural representations
• Closed-form ℓ_r norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓ_p bias
• The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
• Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks
• SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training
• AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent
• Statistical Adaptive Stochastic Gradient Methods
• Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning

Related

• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Generalization in quasi-periodic environments
• Machine learning in and out of equilibrium
• Noise Balance and Stationary Distribution of Stochastic Gradient Descent
• Computing Nonequilibrium Responses with Score-shifted Stochastic Differential Equations