2025/04/17 by Ruinan Jin, Difei Cheng, Jin, Ruinan +9
Computer Science · Engineering · #Stochastic Gradient Optimization Techniques #Privacy-Preserving Technologies in Data #Sparse and Compressive Sensing Techniques
paper · pdf · doi:10.48550/arxiv.2504.12601
Stochastic Gradient Descent (SGD) is widely used in machine learning research. Previous convergence analyses of SGD under the vanishing step-size setting typically require Robbins-Monro conditions. However, in practice, a wider variety of step-size schemes are frequently employed, yet existing convergence results remain limited and often rely on strong assumptions. This paper bridges this gap by introducing a novel analytical framework based on a stopping-time method, enabling asymptotic convergence analysis of SGD under more relaxed step-size conditions and weaker assumptions. In the non-convex setting, we prove the almost sure convergence of SGD iterates for step-sizes \ εt \t ≥ 1 satisfying ∑t=1+∞ εt = +∞ and ∑t=1+∞ εtp < +∞ for some p > 2. Compared with previous studies, our analysis eliminates the global Lipschitz continuity assumption on the loss function and relaxes the boundedness requirements for higher-order moments of stochastic gradients. Building upon the almost sure convergence results, we further establish L2 convergence. These significantly relaxed assumptions make our theoretical results more general, thereby enhancing their applicability in practical scenarios.