Accelerated Gradient Descent by Concatenation of Stepsize Schedules

2024/10/16 by Zhang, Zehao, Jiang, Rujun · 2 citations
#FOS: Mathematics #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2410.12395

Abstract

This work considers stepsize schedules for gradient descent on smooth convex objectives. We extend the existing literature and propose a unified technique for constructing stepsizes with analytic bounds for arbitrary iterations. This technique constructs new stepsize schedules by concatenating two short stepsize schedules. Using this approach, we introduce two new families of stepsize schedules, achieving a convergence rate of O(n^{-log₂(√ 2+1)}) with state-of-the-art constants for the objective value and gradient norm of the last iterate, respectively. Furthermore, our analytically derived stepsize schedules either match or surpass the existing best numerically computed stepsize schedules.

Cited by

• Generalization of Silver Stepsize Schedule to Stochastic Optimization
• Optimized methods for composite optimization: a reduction perspective

Related

• Composing Optimized Stepsize Schedules for Gradient Descent
• Open Problem: Anytime Convergence Rate of Gradient Descent
• Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
• Accelerating Proximal Gradient Descent via Silver Stepsizes
• Generalization of Silver Stepsize Schedule to Stochastic Optimization