Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent

2023/02/14 by Gess, Benjamin, Kassing, Sebastian, Konarovskyi, Vitalii · 4 citations
#46G05 #60G57 #60H15 #68T07 #Analysis of PDEs (math.AP) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Primary 60J05 #Probability (math.PR) #Secondary 60G46

paper · doi:10.48550/arxiv.2302.07125

Abstract

We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.

Cited by

• ODE approximation for the Adam algorithm: General and overparametrized setting
• Effective continuous equations for adaptive SGD: a stochastic analysis view
• Krylov-Veretennikov desomposition for measure-valued processes induced by SDEs with interaction on Riemannian manifolds
• Strong Feller Regularisation of 1-d Nonlinear Transport by Reflected Ornstein-Uhlenbeck Noise

Related

• Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
• Effective continuous equations for adaptive SGD: a stochastic analysis view
• Stochastic Normalizing Flows
• Rigorous dynamical mean field theory for stochastic gradient descent methods
• Modified Equations for Stochastic Optimization