Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

2017/06/17 by Mahesh Chandra Mukkamala, Matthias Hein, Mukkamala, Mahesh Chandra +1 · 4 citations
Computer Science · Decision Sciences · Engineering · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1706.05507

openalex publication_date 2017/06/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Adaptive gradient methods have become recently very popular, in particular as they have been shown to be useful in the training of deep neural networks. In this paper we have analyzed RMSProp, originally proposed for the training of deep neural networks, in the context of online convex optimization and show √(T)-type regret bounds. Moreover, we propose two variants SC-Adagrad and SC-RMSProp for which we show logarithmic regret bounds for strongly convex functions. Finally, we demonstrate in the experiments that these new variants outperform other adaptive gradient techniques or stochastic gradient descent in the optimization of strongly convex functions as well as in training of deep neural networks.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Deep Learning in Neural Networks: An Overview
• An overview of gradient descent optimization algorithms
• Rethinking the Inception Architecture for Computer Vision
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization
• Optimizing the Adversarial Perturbation with a Momentum-based Adaptive Matrix
• Adaptivity via a Parallel Architecture for Stochastic Gradient Methods
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Related

• Dynamic Regret of Adaptive Gradient Methods for Strongly Convex Problems
• SAdam: A Variant of Adam for Strongly Convex Functions
• Less Regret via Online Conditioning
• Gradient-Variation Regret Bounds for Unconstrained Online Learning
• Adaptivity and Optimality: A Universal Algorithm for Online Convex Optimization