Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization

2019/11/19 by Dvinskikh, Darina, Ogaltsov, Aleksandr, Gasnikov, Alexander +3
#FOS: Mathematics #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.1911.08380

Abstract

In this paper we propose several adaptive gradient methods for stochastic optimization. Unlike AdaGrad-type of methods, our algorithms are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. We consider an accelerated and non-accelerated gradient descent for convex problems and gradient descent for non-convex problems. In the experiments we demonstrate superiority of our methods to existing adaptive methods, e.g. AdaGrad and Adam.

Related

• Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications
• Adaptive First-and Zeroth-order Methods for Weakly Convex Stochastic Optimization Problems
• Stochastic Adaptive Gradient Descent Without Descent
• Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization
• Adaptive fast gradient method in stochastic optimization tasks