2024/08/03 by Saeed Masiha, Saber Salehkaleybar, Masiha, Saeed +7
Computer Science · Engineering · Mathematics · #Advanced Memory and Neural Computing #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2408.01839
openalex publication_date 2024/08/03 · openalex created_date 2025/01/17 · openalex updated_date 2026/07/28
We study the local convergence rate of stochastic first-order methods under a local α-Polyak-Lojasiewicz (α-PL) condition in a neighborhood of a target connected component M of the local minimizer set. The parameter α∈ [1,2] is the exponent of the gradient norm in the α-PL inequality: α=2 recovers the classical PL case, α=1 corresponds to Holder-type error bounds, and intermediate values interpolate between these regimes. Our performance criterion is the number of oracle queries required to output x with F(x)-l ≤ ε, where l := F(y) for any y ∈ M. We work in a local regime where the algorithm is initialized near M and, with high probability, its iterates remain in that neighborhood. We establish a lower bound Ω(ε-2/α) for all stochastic first-order methods in this regime, and we obtain a matching upper bound O(ε-2/α) for 1 ≤ α< 2 via a SARAH-type variance-reduced method with time-varying batch sizes and step sizes. In the convex setting, assuming a local α-PL condition on the ε-sublevel set, we further show a complexity lower bound \widetildeΩ(ε-2/α) for reaching an ε-global optimum, matching the ε-dependence of known accelerated stochastic subgradient methods.