The Two Regimes of Deep Network Training

2020/02/24 by Guillaume Leclerc, Aleksander Mądry, Aleksander Madry +2 · 4 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.10376

14 pages (5 of appendix), 14 figures

arxiv created 2020/02/24 · openalex publication_date 2020/02/24 · arxiv updated 2020/02/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning rate schedule has a major impact on the performance of deep learning models. Still, the choice of a schedule is often heuristical. We aim to develop a precise understanding of the effects of different learning rate schedules and the appropriate way to select them. To this end, we isolate two distinct phases of training, the first, which we refer to as the "large-step" regime, exhibits a rather poor performance from an optimization point of view but is the primary contributor to model generalization; the latter, "small-step" regime exhibits much more "convex-like" optimization behavior but used in isolation produces models that generalize poorly. We find that by treating these regimes separately-and em specializing our training algorithm to each one of them, we can significantly simplify learning rate schedules.

Citations

• On the limited memory BFGS method for large scale optimization
• ImageNet Large Scale Visual Recognition Challenge
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• An Exponential Learning Rate Schedule for Deep Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Deep Progressive Training: scaling up depth capacity of zero/one-layer models
• Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysis
• Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate
• Implicit bias of deep linear networks in the large learning rate phase

Related

• Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minima
• Disentangling Adaptive Gradient Methods from Learning Rates
• The large learning rate phase of deep learning: the catapult mechanism
• The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
• LRTuner: A Learning Rate Tuner for Deep Neural Networks