Normalized Gradients for All

2023/08/10 by Orabona, Francesco · 7 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2308.05621

Abstract

In this short note, I show how to adapt to Hölder smoothness using normalized gradients in a black-box way. Moreover, the bound will depend on a novel notion of local Hölder smoothness. The main idea directly comes from Levy [2017].

Cited by

• New Results on the Polyak Stepsize: Tight Convergence Analysis and Universal Function Classes
• Gradient-Variation Online Adaptivity for Accelerated Optimization with Hölder Smoothness
• Accelerated Distance-adaptive Methods for Hölder Smooth and Convex Optimization
• Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimization
• Stochastic Adaptive Gradient Descent Without Descent
• AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates
• Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimization

Related

• Near-optimal learning with average Hölder smoothness
• General Holder Smooth Convergence Rates Follow From Specialized Rates Assuming Growth Bounds
• The Power of Normalization: Faster Evasion of Saddle Points
• Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity
• (Adaptive) Scaled gradient methods beyond locally Holder smoothness: Lyapunov analysis, convergence rate and complexity