Gradient descent with momentum --- to accelerate or to super-accelerate?

2020/01/17 by Goran Nakerst, John Brennan, Nakerst, Goran +3 · 5 citations
Computer Science · Mathematics · #Acceleration #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Classical mechanics #Computer science #FOS: Computer and information sciences #FOS: Mathematics #Function (biology) #Geometry #Gradient descent #Hyperparameter #MNIST database #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical optimization #Mathematics #Minification #Momentum (technical analysis) #Neural Networks and Applications #Optimization and Control (math.OC) #Physics #Position (finance) #Quadratic equation #Stochastic Gradient Optimization Techniques #cs.LG #math.OC #stat.ML

paper · pdf · doi:10.48550/arxiv.2001.06472

published in arXiv (Cornell University) (Cornell University) · 19 pages + references, 8 figures. A variant of Nesterov acceleration is proposed and studied

arxiv created 2020/01/17 · openalex publication_date 2020/01/17 · arxiv updated 2020/01/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We consider gradient descent with `momentum', a widely used method for loss function minimization in machine learning. This method is often used with `Nesterov acceleration', meaning that the gradient is evaluated not at the current position in parameter space, but at the estimated position after one step. In this work, we show that the algorithm can be improved by extending this `acceleration' --- by using the gradient at an estimated position several steps ahead rather than just one step ahead. How far one looks ahead in this `super-acceleration' algorithm is determined by a new hyperparameter. Considering a one-parameter quadratic loss function, the optimal value of the super-acceleration can be exactly calculated and analytically estimated. We show explicitly that super-accelerating the momentum algorithm is beneficial, not only for this idealized problem, but also for several synthetic loss landscapes and for the MNIST classification task with neural networks. Super-acceleration is also easy to incorporate into adaptive algorithms like RMSProp or Adam, and is shown to improve these algorithms.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Learning representations by back-propagating errors
• An overview of gradient descent optimization algorithms
• A Lyapunov Analysis of Momentum Methods in Optimization
• On the Convergence of Adam and Beyond
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• Deep learning
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Understanding the Role of Momentum in Stochastic Gradient Methods
• On Empirical Comparisons of Optimizers for Deep Learning

Cited by

• Preconditioning Natural and Second Order Gradient Descent in Quantum Optimization: A Performance Benchmark

Related

• ADINE: An Adaptive Momentum Method for Stochastic Gradient Descent
• A New Accelerated Stochastic Gradient Method with Momentum
• The Marginal Value of Momentum for Small Learning Rate SGD
• Asymmetric Momentum: A Rethinking of Gradient Descent
• Understanding the Role of Momentum in Stochastic Gradient Methods