Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate\n Schedule

2020/03/09 by Nikhil Iyer, V Thejas, Iyer, Nikhil +7 · 3 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2003.03977

openalex publication_date 2020/03/09 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

Several papers argue that wide minima generalize better than narrow minima.\nIn this paper, through detailed experiments that not only corroborate the\ngeneralization properties of wide minima, we also provide empirical evidence\nfor a new hypothesis that the density of wide minima is likely lower than the\ndensity of narrow minima. Further, motivated by this hypothesis, we design a\nnovel explore-exploit learning rate schedule. On a variety of image and natural\nlanguage datasets, compared to their original hand-tuned learning rate\nbaselines, we show that our explore-exploit schedule can result in either up to\n0.84% higher absolute accuracy using the original training budget or up to 57%\nreduced training time while achieving the original reported accuracy. For\nexample, we achieve state-of-the-art (SOTA) accuracy for IWSLT'14 (DE-EN)\ndataset by just modifying the learning rate schedule of a high performing\nmodel.\n

Citations

• Adam: A Method for Stochastic Optimization
• ImageNet Large Scale Visual Recognition Challenge
• SGDR: Stochastic Gradient Descent with Warm Restarts
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Sharp Minima Can Generalize For Deep Nets
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Three Factors Influencing Minima in SGD
• Stronger generalization bounds for deep nets via a compression approach
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Identifying Generalization Properties in Neural Networks
• Information Geometry of Orthogonal Initializations and Training
• An Empirical Model of Large-Batch Training
• Flat Minima
• On the Variance of the Adaptive Learning Rate and Beyond
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Mid-Training of Large Language Models: A Survey
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
• Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler

Related

• Large Learning Rates Improve Generalization: But How Large Are We Talking About?
• Tuning Learning Rates with the Cumulative-Learning Constant
• Where Do Large Learning Rates Lead Us?
• Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models
• Using More Data to Speed-up Training Time