Conjugate-gradient-based Adam for stochastic optimization and its application to deep learning

2020/02/29 by Yu Kobayashi, Kobayashi, Yu, Hideaki Iiduka +1
Computer Science · Mathematics · #Advanced Optimization Algorithms Research #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Neural Networks and Applications #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2003.00231

openalex publication_date 2020/02/29 · openalex created_date 2020/03/06 · openalex updated_date 2026/07/28

Abstract

This paper proposes a conjugate-gradient-based Adam algorithm blending Adam with nonlinear conjugate gradient methods and shows its convergence analysis. Numerical experiments on text classification and image classification show that the proposed algorithm can train deep neural network models in fewer epochs than the existing adaptive stochastic optimization algorithms can.

Citations

• Stochastic Approximation: A Dynamical Systems Viewpoint
• A Stochastic Approximation Method
• Function minimization by conjugate gradients
• On the Convergence of Adam and Beyond

Related

• Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates
• Adam⁺: A Stochastic Method with Adaptive Variance Reduction
• Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning
• Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks
• Optimization Methods in Deep Learning: A Comprehensive Overview