2024/07/05 by Antonio Orvieto, Lin Xiao, Orvieto, Antonio +1 · 2 voices · 3 citations
Computer Science · Engineering · Mathematics · #Advanced Image Processing Techniques #Advanced Numerical Analysis Techniques #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques #cs.LG #math.OC
paper · pdf · doi:10.48550/arxiv.2407.04358
openalex publication_date 2024/07/05 · arxiv published 2024/07/05 · arxiv updated 2024/07/05 · openalex created_date 2024/07/09 · openalex updated_date 2026/07/28
We consider the problem of minimizing the average of a large number of smooth but possibly non-convex functions. In the context of most machine learning applications, each loss function is non-negative and thus can be expressed as the composition of a square and its real-valued square root. This reformulation allows us to apply the Gauss-Newton method, or the Levenberg-Marquardt method when adding a quadratic regularization. The resulting algorithm, while being computationally as efficient as the vanilla stochastic gradient method, is highly adaptive and can automatically warmup and decay the effective stepsize while tracking the non-negative loss landscape. We provide a tight convergence analysis, leveraging new techniques, in the stochastic convex and non-convex settings. In particular, in the convex case, the method does not require access to the gradient Lipshitz constant for convergence, and is guaranteed to never diverge. The convergence rates and empirical evaluations compare favorably to the classical (stochastic) gradient method as well as to several other adaptive methods.