Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent

2016/07/07 by Aleksandar Botev, Guy Lever, Botev, Aleksandar +4 · 3 citations
Computer Science · Engineering · Mathematics · #Advanced Optimization Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1607.01981

openalex publication_date 2016/07/07 · arxiv created 2016/07/11 · arxiv updated 2016/07/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we term Regularised Gradient Descent, can converge more quickly than either Nesterov's algorithm or the classical momentum algorithm.

Cited by

• A Proportional-Integral Controller-Incorporated SGD Algorithm for High Efficient Latent Factor Analysis
• Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis
• DATA: Multi-Disentanglement based Contrastive Learning for Open-World Semi-Supervised Deepfake Attribution

Related

• Gradient descent with momentum --- to accelerate or to super-accelerate?
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Point Convergence of Nesterov's Accelerated Gradient Method: An AI-Assisted Proof
• A Family of Controllable Momentum Coefficients for Forward-Backward Accelerated Algorithms
• Accelerating Proximal Gradient-type Algorithms using Damped Anderson Acceleration with Restarts and Nesterov Initialization