Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization

2020/10/01 by Defazio, Aaron · 5 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2010.00406

Abstract

Momentum methods are now used pervasively within the machine learning community for training non-convex models such as deep neural networks. Empirically, they out perform traditional stochastic gradient descent (SGD) approaches. In this work we develop a Lyapunov analysis of SGD with momentum (SGD+M), by utilizing a equivalent rewriting of the method known as the stochastic primal averaging (SPA) form. This analysis is much tighter than previous theory in the non-convex case, and due to this we are able to give precise insights into when SGD+M may out-perform SGD, and what hyper-parameter schedules will work and why.

Cited by

• Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs
• SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradients
• Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization
• Analysis of Schedule-Free Nonconvex Optimization
• Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis

Related

• Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality
• Random Scaling and Momentum for Non-smooth Non-convex Optimization
• The Marginal Value of Momentum for Small Learning Rate SGD
• Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum
• Escaping Saddle Points Faster with Stochastic Momentum