Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method

2022/06/14 by Aaron Defazio, Defazio, Aaron, Baoyu Zhou +3 · 3 citations
Computer Science · Engineering · #Stochastic Gradient Optimization Techniques #Neural Networks and Applications #Advanced Adaptive Filtering Techniques

paper · pdf · doi:10.48550/arxiv.2206.06900

Abstract

The classical AdaGrad method adapts the learning rate by dividing by the square root of a sum of squared gradients. Because this sum on the denominator is increasing, the method can only decrease step sizes over time, and requires a learning rate scaling hyper-parameter to be carefully tuned. To overcome this restriction, we introduce GradaGrad, a method in the same family that naturally grows or shrinks the learning rate based on a different accumulation in the denominator, one that can both increase and decrease. We show that it obeys a similar convergence rate as AdaGrad and demonstrate its non-monotone adaptation capability with experiments.

Cited by

• Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization
• Stochastic Adaptive Gradient Descent Without Descent
• Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimization

Related

• Adaptive Learning Rates for Faster Stochastic Gradient Methods
• The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance
• Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
• On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes
• AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations