Towards faster first order methods: A continuous-time model to interpolate between speed and function value restart

2025/06/12 by Juan José Maulén, Maulén, Juan José, Juan Peypouquet +2
Computer Science · #FOS: Mathematics #Neural Networks and Applications #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2506.11267

openalex publication_date 2025/06/12 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

We introduce a new restarting scheme for a continuous inertial dynamics with Hessian driven-damping, and establish a linear convergence rate for the function values along the restarted trajectories. The proposed routine is implemented without knowing the strong convexity parameter, and is a generalization of existing speed restart schemes. It interpolates between speed and function value restarts, considerably delaying the restarting time, while preserving convergence and function value decrease. Numerical experiments show an improvement in the convergence rates for both continuous-time dynamical systems, and the associated accelerated first-order algorithms derived via time discretization.

Citations

• Inertial Quadratic Majorization Minimization with Application to Kernel Regularized Learning
• Fast convex optimization via inertial systems with asymptotically vanishing viscosity and Hessian-driven damping
• Accelerated Gradient Methods via Inertial Systems with Hessian-driven Damping
• An accelerated gradient method with adaptive restart for convex multiobjective optimization problems
• A Speed Restart Scheme for a Dynamical System with Hessian-Driven Damping and Three Constant Coefficients
• Accelerated Gradient Methods with Gradient Restart: Global Linear Convergence
• Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity
• A speed restart scheme for a dynamics with Hessian driven damping
• Linear Convergence of ISTA and FISTA
• Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the O(ε^-7/4) Complexity
• Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent
• First-order optimization algorithms via inertial systems with Hessian\n driven damping
• Restart FISTA with Global Linear Convergence
• Understanding the Acceleration Phenomenon via High-Resolution Differential Equations
• Fast convex optimization via inertial dynamics with Hessian driven\n damping
• Asymptotic for a second order evolution equation with convex potential\n and vanishing damping term
• Linear convergence of first order methods for non-strongly convex optimization
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• Adaptive Restart for Accelerated Gradient Schemes

Related

• A Speed Restart Scheme for a Dynamical System with Hessian-Driven Damping and Three Constant Coefficients
• A speed restart scheme for a dynamics with Hessian driven damping
• Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping
• First-order optimization algorithms via inertial systems with Hessian\n driven damping
• Accelerated Gradient Methods via Inertial Systems with Hessian-driven Damping