2020/12/07 by Anas Barakat, A. Barakat, Barakat, A. +9 · 3 citations
Computer Science · Engineering · Mathematics · #34A12 #60F99 #62L20 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Probability (math.PR) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #math.OC #math.PR #msc:34A12 #msc:60F99 #msc:62L20 #stat.ML
paper · pdf · doi:10.48550/arxiv.2012.04002
Accepted for publication in Electronic Journal of Statistics. 49 pages
openalex publication_date 2020/12/07 · arxiv created 2021/07/10 · arxiv updated 2021/07/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this paper, a general stochastic optimization procedure is studied, unifying several variants of the stochastic gradient descent such as, among others, the stochastic heavy ball method, the Stochastic Nesterov Accelerated Gradient algorithm (S-NAG), and the widely used Adam algorithm. The algorithm is seen as a noisy Euler discretization of a non-autonomous ordinary differential equation, recently introduced by Belotto da Silva and Gazeau, which is analyzed in depth. Assuming that the objective function is non-convex and differentiable, the stability and the almost sure convergence of the iterates to the set of critical points are established. A noteworthy special case is the convergence proof of S-NAG in a non-convex setting. Under some assumptions, the convergence rate is provided under the form of a Central Limit Theorem. Finally, the non-convergence of the algorithm to undesired critical points, such as local maxima or saddle points, is established. Here, the main ingredient is a new avoidance of traps result for non-autonomous settings, which is of independent interest.