Fluctuation-dissipation relations for stochastic gradient descent

2018/09/28 by Sho Yaida, Yaida, Sho · 5 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1810.00004

openalex publication_date 2018/09/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm. These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule. We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity. Our claims are empirically verified.

Citations

• A Stochastic Approximation Method
• Statistical-Mechanical Theory of Irreversible Processes. I. General Theory and Simple Applications to Magnetic and Conduction Problems
• Über die von der molekularkinetischen Theorie der Wärme geforderte Bewegung von in ruhenden Flüssigkeiten suspendierten Teilchen
• Gradient-based learning applied to document recognition
• Three Factors Influencing Minima in SGD
• Learning Multiple Layers of Features from Tiny Images

Cited by

• The Universe Learning Itself: On the Evolution of Dynamics from the Big Bang to Machine Intelligence
• Contribution of task-irrelevant stimuli to drift of neural representations
• Closed-form ℓ_r norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓ_p bias
• The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
• Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks

Related

• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Generalization in quasi-periodic environments
• Machine learning in and out of equilibrium
• Noise Balance and Stationary Distribution of Stochastic Gradient Descent
• Computing Nonequilibrium Responses with Score-shifted Stochastic Differential Equations