Disentangling Adaptive Gradient Methods from Learning Rates

2020/02/26 by Naman Agarwal, Agarwal, Naman, Rohan Anil +7 · 4 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2002.11803

openalex publication_date 2020/02/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We investigate several confounding factors in the evaluation of optimization algorithms for deep learning. Primarily, we take a deeper look at how adaptive gradient methods interact with the learning rate schedule, a notoriously difficult-to-tune hyperparameter which has dramatic effects on the convergence and generalization of neural network training. We introduce a "grafting" experiment which decouples an update's magnitude from its direction, finding that many existing beliefs in the literature may have arisen from insufficient isolation of the implicit schedule of step sizes. Alongside this contribution, we present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods, aimed at bringing more clarity to this space.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On the difficulty of training Recurrent Neural Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Fine-Grained Analysis of Optimization and Generalization for\n Overparameterized Two-Layer Neural Networks
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Memory-Efficient Adaptive Optimization
• An Exponential Learning Rate Schedule for Deep Learning

Cited by

• Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
• Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

Related

• Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• LipschitzLR: Using theoretically computed adaptive learning rates for fast convergence
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• AdaS: Adaptive Scheduling of Stochastic Gradients