Parameter-free version of Adaptive Gradient Methods for Strongly-Convex Functions

2023/06/11 by Deepak Gouda, Gouda, Deepak, Hassan Naveed +3
Computer Science · Engineering · Decision Sciences · #Stochastic Gradient Optimization Techniques #Sparse and Compressive Sensing Techniques #Advanced Bandit Algorithms Research

paper · pdf · doi:10.48550/arxiv.2306.06613

Abstract

The optimal learning rate for adaptive gradient methods applied to λ-strongly convex functions relies on the parameters λ and learning rate η. In this paper, we adapt a universal algorithm along the lines of Metagrad, to get rid of this dependence on λ and η. The main idea is to concurrently run multiple experts and combine their predictions to a master algorithm. This master enjoys O(d log T) regret bounds.

Related

• Fast adaptive by constants of strong-convexity and Lipschitz for\n gradient first order methods
• Polyak Steps for Adaptive Fast Gradient Method
• Dual Adaptivity: A Universal Algorithm for Minimizing the Adaptive Regret of Convex Functions
• Adaptive Gradient Methods for Constrained Convex Optimization and\n Variational Inequalities
• Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)