A Variational Perspective on High-Resolution ODEs

2023/11/03 by Maskan, Hoomaan, Zygalakis, Konstantinos C., Yurtsever, Alp · 2 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Numerical Analysis (math.NA) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2311.02002

Abstract

We consider unconstrained minimization of smooth convex functions. We propose a novel variational perspective using forced Euler-Lagrange equation that allows for studying high-resolution ODEs. Through this, we obtain a faster convergence rate for gradient norm minimization using Nesterov's accelerated gradient method. Additionally, we show that Nesterov's method can be interpreted as a rate-matching discretization of an appropriately chosen high-resolution ODE. Finally, using the results from the new variational perspective, we propose a stochastic method for noisy gradients. Several numerical experiments compare and illustrate our stochastic algorithm with state of the art methods.

Cited by

• A √(2)-accelerated FISTA for composite strongly convex problems
• Stochastic Quantum Hamiltonian Descent

Related

• Understanding the Acceleration Phenomenon via High-Resolution Differential Equations
• Unifying Nesterov's Accelerated Gradient Methods for Convex and Strongly Convex Objective Functions: From Continuous-Time Dynamics to Discrete-Time Algorithms
• Optimal convergence rates for Nesterov acceleration
• Understanding Accelerated Gradient Methods: Lyapunov Analyses and Hamiltonian Assisted Interpretations
• Accelerated optimization algorithms and ordinary differential equations: the convex non Euclidean case