AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent

2025/05/27 by Nikola Surjanovic, Surjanovic, Nikola, Alexandre Bouchard‐Côté +3 · 1 citation
Computer Science · #Advanced Neural Network Applications #Computation (stat.CO) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.21651

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks.

Citations

• The Road Less Scheduled
• Accelerated Parameter-Free Stochastic Optimization
• Tuning-Free Stochastic Optimization
• How Free is Parameter-Free Stochastic Optimization?
• Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?
• Gradient Descent Converges Linearly for Logistic Regression on Separable Data
• Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models
• Prodigy: An Expeditiously Adaptive Parameter-Free Learner
• DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method
• DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule
• Handbook of Convergence Theorems for (Stochastic) Gradient Methods
• Learning-Rate-Free Learning by D-Adaptation
• Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution
• Making SGD Parameter-Free
• A Framework for Improving the Reliability of Black-box Variational Inference
• Variance-Reduced Methods for Machine Learning
• Robust, Accurate Stochastic Optimization for Variational Inference
• Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum
• On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent
• Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast\n Convergence
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Using Statistics to Automate Stochastic Optimization
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
• Anytime Tail Averaging
• Beating SGD Saturation with Tail-Averaging and Minibatching
• Fluctuation-dissipation relations for stochastic gradient descent
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Convergence diagnostics for stochastic gradient descent with constant step size
• Sticking the Landing: Simple, Lower-Variance Gradient Estimators for\n Variational Inference
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Deep Residual Learning for Image Recognition
• Julia: A Fresh Approach to Numerical Computing
• Julia: A Fresh Approach to Numerical Computing
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence\n Results and Optimal Averaging Schemes
• Making Gradient Descent Optimal for Strongly Convex Stochastic\n Optimization
• Inference from Iterative Simulation Using Multiple Sequences
• A Stochastic Approximation Method

Cited by

• AutoGD: Automatic Learning Rate Selection for Gradient Descent

Related

• AutoGD: Automatic Learning Rate Selection for Gradient Descent
• Explainable Learning Rate Regimes for Stochastic Optimization
• No More Pesky Learning Rates
• Statistical Adaptive Stochastic Gradient Methods
• Using Statistics to Automate Stochastic Optimization