Equilibrated adaptive learning rates for non-convex optimization

2015/02/15 by Yann Dauphin, Dauphin, Yann N., Harm de Vries +3 · 10 citations
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning and ELM #Numerical Analysis (math.NA) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1502.04390

openalex publication_date 2015/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the so-called equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Matrix-Free Approximate Equilibration
• Theano: new features and speed improvements
• Krylov Subspace Descent for Deep Learning
• Unit Tests for Stochastic Optimization

Cited by

• Optimizing Rank for High-Fidelity Implicit Neural Representations
• Training Across Reservoirs: Using Numerical Differentiation To Couple Trainable Networks With Black-Box Reservoirs
• Stochastic Trace and Diagonal Estimator for Tensors
• PrecoG: an efficient unitary split preconditioner for the transform-domain LMS filter via graph Laplacian regularization
• Single-Solution Hypervolume Maximization and its use for Improving Generalization of Neural Networks
• DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization
• Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
• Preconditioned Stochastic Gradient Descent
• Handwritten isolated Bangla compound character recognition: A new benchmark using a novel deep learning approach
• Robust and efficient algorithms for high-dimensional black-box quantum optimization

Related

• Adaptive Strategies in Non-convex Optimization
• Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks
• ADLER -- An efficient Hessian-based strategy for adaptive learning rate
• Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
• Adaptive Gradient Methods at the Edge of Stability