Accelerated Gradient Flow: Risk, Stability, and Implicit Regularization

2022/01/20 by Sheng Yue, Sheng, Yue, Alnur Ali +1
Computer Science · Engineering · Mathematics · #Stochastic Gradient Optimization Techniques #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2201.08311

Abstract

Acceleration and momentum are the de facto standard in modern applications of machine learning and optimization, yet the bulk of the work on implicit regularization focuses instead on unaccelerated methods. In this paper, we study the statistical risk of the iterates generated by Nesterov's accelerated gradient method and Polyak's heavy ball method, when applied to least squares regression, drawing several connections to explicit penalization. We carry out our analyses in continuous-time, allowing us to make sharper statements than in prior work, and revealing complex interactions between early stopping, stability, and the curvature of the loss function.

Related

• The Implicit Regularization of Momentum Gradient Descent with Early Stopping
• Gradient descent follows the regularization path for general losses
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• Regularized Risk Minimization by Nesterov's Accelerated Gradient Methods: Algorithmic Extensions and Empirical Studies
• A Continuous-Time View of Early Stopping for Least Squares