On uniform-in-time diffusion approximation for stochastic gradient descent

2022/07/11 by Lei Li, Yuliang Wang, Li, Lei +1 · 1 citation
Economics, Econometrics and Finance · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Mathematical Biology Tumor Growth #Stochastic processes and financial applications

paper · pdf · doi:10.48550/arxiv.2207.04922

openalex publication_date 2022/07/11 · openalex created_date 2022/07/13 · openalex updated_date 2026/07/28

Abstract

The diffusion approximation of stochastic gradient descent (SGD) in current literature is only valid on a finite time interval. In this paper, we establish the uniform-in-time diffusion approximation of SGD, by only assuming that the expected loss is strongly convex and some other mild conditions, without assuming the convexity of each random loss function. The main technique is to establish the exponential decay rates of the derivatives of the solution to the backward Kolmogorov equation. The uniform-in-time approximation allows us to study asymptotic behaviors of SGD via the continuous stochastic differential equation (SDE) even when the random objective function f(⋅;ξ) is not strongly convex.

Cited by

• Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD

Related

• Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation
• Convergence rates and approximation results for SGD and its continuous-time counterpart
• Uniform in time estimates for the weak error of the Euler method for SDEs and a Pathwise Approach to Derivative Estimates for Diffusion Semigroups
• Stochastic Gradient Descent in Continuous Time: A Central Limit Theorem
• Uniform in time convergence of numerical schemes for stochastic differential equations via Strong Exponential stability: Euler methods, Split-Step and Tamed Schemes