Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum

2020/08/27 by Jerry Chee, Ping Li, Chee, Jerry +1 · 1 citation
Computer Science · Engineering · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2008.12224

openalex publication_date 2020/08/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Convergence detection of iterative stochastic optimization methods is of great practical interest. This paper considers stochastic gradient descent (SGD) with a constant learning rate and momentum. We show that there exists a transient phase in which iterates move towards a region of interest, and a stationary phase in which iterates remain bounded in that region around a minimum point. We construct a statistical diagnostic test for convergence to the stationary phase using the inner product between successive gradients and demonstrate that the proposed diagnostic works well. We theoretically and empirically characterize how momentum can affect the test statistic of the diagnostic, and how the test statistic captures a relatively sparse signal within the gradients in convergence. Finally, we demonstrate an application to automatically tune the learning rate by reducing it each time stationarity is detected, and show the procedure is robust to mis-specified initial rates.

Citations

• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• Improving Generalization Performance by Switching from Adam to SGD
• Uncertainty Quantification for Online Learning and Stochastic Approximation via Hierarchical Incremental Gradient Descent

Cited by

• AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent

Related

• Convergence diagnostics for stochastic gradient descent with constant step size
• On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent
• First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms
• Convergence rates of stochastic gradient method with independent sequences of step-size and momentum weight
• An Improved Analysis of Stochastic Gradient Descent with Momentum