Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration

2018/07/18 by Soham De, De, Soham, Anirbit Mukherjee +3 · 1 citation
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1807.06766

openalex publication_date 2018/07/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the running time. Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov's Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter β₁. We show that at very high values of the momentum parameter (β₁ = 0.99) ADAM outperforms a carefully tuned NAG on most of our experiments, in terms of getting lower training and test losses. On the other hand, NAG can sometimes do better when ADAM's β₁ is set to the most commonly used value: β₁ = 0.9, indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance. We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms, and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Why Regularized Auto-Encoders learn Sparse Representation?
• DRAW: A Recurrent Neural Network For Image Generation
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• Training Deep AutoEncoders for Collaborative Filtering
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly\n Convex Composite Objectives
• On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

Cited by

• Globally aware optimization with resurgence

Related

• A Sufficient Condition for Convergences of Adam and RMSProp
• A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings
• On the Convergence of Adam and Beyond
• Sharp higher order convergence rates for the Adam optimizer
• Convergence of the Iterates for Momentum and RMSProp for Local Smooth Functions: Adaptation is the Key