Nesterov's Accelerated Gradient and Momentum as approximations to\n Regularised Update Descent

2016/07/07 by Aleksandar Botev, Botev, Aleksandar, Guy Lever +3 · 3 citations
Computer Science · Engineering · Mathematics · #Advanced Optimization Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1607.01981

openalex publication_date 2016/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a unifying framework for adapting the update direction in\ngradient-based iterative optimization methods. As natural special cases we\nre-derive classical momentum and Nesterov's accelerated gradient method,\nlending a new intuitive interpretation to the latter algorithm. We show that a\nnew algorithm, which we term Regularised Gradient Descent, can converge more\nquickly than either Nesterov's algorithm or the classical momentum algorithm.\n

Cited by

• A Proportional-Integral Controller-Incorporated SGD Algorithm for High Efficient Latent Factor Analysis
• Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis
• DATA: Multi-Disentanglement based Contrastive Learning for Open-World Semi-Supervised Deepfake Attribution

Related

• Gradient descent with momentum --- to accelerate or to super-accelerate?
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Point Convergence of Nesterov's Accelerated Gradient Method: An AI-Assisted Proof
• A Family of Controllable Momentum Coefficients for Forward-Backward Accelerated Algorithms
• Accelerating Proximal Gradient-type Algorithms using Damped Anderson Acceleration with Restarts and Nesterov Initialization