Revisiting the central limit theorems for the SGD-type methods

2022/07/24 by Tiejun Li, Li, Tiejun, Tiannan Xiao +3 · 2 citations
Computer Science · Economics, Econometrics and Finance · Mathematics · #37N40 #60F05 #60J22 #FOS: Mathematics #Optimization and Control (math.OC) #Probability (math.PR) #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques #Stochastic processes and financial applications

paper · pdf · doi:10.48550/arxiv.2207.11755

openalex publication_date 2022/07/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We revisited the central limit theorem (CLT) for stochastic gradient descent (SGD) type methods, including the vanilla SGD, momentum SGD and Nesterov accelerated SGD methods with constant or vanishing damping parameters. By taking advantage of Lyapunov function technique and L^p bound estimates, we established the CLT under more general conditions on learning rates for broader classes of SGD methods compared with previous results. The CLT for the time average was also investigated, and we found that it held in the linear case, while it was not generally true in nonlinear situation. Numerical tests were also carried out to verify our theoretical analysis.

Cited by

• Functional central limit theorem for Euler--Maruyama scheme with decreasing step sizes
• Stochastic Quadrature Rules for Solving PDEs using Neural Networks

Related

• A convergence study of SGD-type methods for stochastic optimization
• Stochastic Gradient Descent in Continuous Time: A Central Limit Theorem
• First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms
• Effective continuous equations for adaptive SGD: a stochastic analysis view
• New Tight Bounds for SGD without Variance Assumption: A Computer-Aided Lyapunov Analysis