Convergence and Convergence Rate of Stochastic Gradient Search in the\n Case of Multiple and Non-Isolated Extrema

2009/07/06 by Vladislav B. Tadić, Tadic, Vladislav B. · 1 citation
Computer Science · Economics, Econometrics and Finance · Mathematics · #62L20 #90C15 #93E12 #93E35 #Advanced Optimization Algorithms Research #FOS: Mathematics #Optimization and Control (math.OC) #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques #Stochastic processes and financial applications

paper · pdf · doi:10.48550/arxiv.0907.1020

openalex publication_date 2009/07/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The asymptotic behavior of stochastic gradient algorithms is studied. Relying\non results from differential geometry (Lojasiewicz gradient inequality), the\nsingle limit-point convergence of the algorithm iterates is demonstrated and\nrelatively tight bounds on the convergence rate are derived. In sharp contrast\nto the existing asymptotic results, the new results presented here allow the\nobjective function to have multiple and non-isolated minima. The new results\nalso offer new insights into the asymptotic properties of several classes of\nrecursive algorithms which are routinely used in engineering, statistics,\nmachine learning and operations research.\n

Cited by

• Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization

Related

• Convergence Rate of Stochastic Gradient Search in the Case of Multiple and Non-Isolated Minima
• Asymptotic Bias of Stochastic Gradient Search
• Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms
• On strict convergence of stochastic gradients
• On the Convergence of mSGD and AdaGrad for Stochastic Optimization