Deep Double Descent: Where Bigger Models and More Data Hurt

2019/12/04 by Nakkiran, Preetum, Kaplun, Gal, Bansal, Yamini +3 · 41 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE)

paper · doi:10.48550/arxiv.1912.02292

Abstract

We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.

Cited by

• Continuized Nesterov Acceleration for Non-Convex Optimization
• Distill Video Datasets into Images
• Understanding Overparametrization in Survival Models through Interpolation
• Large and Small Model Collaboration for Air Interface
• On Learning-Curve Monotonicity for Maximum Likelihood Estimators
• Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression
• Parameter Reduction Improves Vision Transformers: A Comparative Study of Sharing and Width Reduction
• Estimating the Effective Rank of Vision Transformers via Low-Rank Factorization
• A3T-GCN for FTSE100 Components Price Forecasting
• Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI
• Minimalist machine-learned interatomic potentials can predict complex structural behaviors accurately
• Improving Quantum Neural Networks exploration by Noise-Induced Equalization
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
• On The Presence of Double-Descent in Deep Reinforcement Learning
• Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers
• Origins and mitigation of double descent in reduced order modeling
• A Theory of Generalization in Deep Learning
• On the Anisotropy of Score-Based Generative Models
• A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation
• C-SWAP: Explainability-Aware Structured Pruning for Efficient Neural Networks Compression
• Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression
• Identifying Subgroup and Context Effects in Conjoint Experiments
• The Effect of Label Noise on the Information Content of Neural Representations
• Hybrid Sequential Quantum Computing
• Double Descent as a Lens for Sample Efficiency in Autoregressive vs. Discrete Diffusion Models
• Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learning
• Closed-form ℓ_r norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓ_p bias
• Simplifying Neural Networks During Training
• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
• \boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning
• Inspired by machine learning optimization: can gradient-based optimizers solve cycle skipping in full waveform inversion given sufficient iterations?
• Data coarse graining can improve model performance
• Deep learning and abstractive summarisation for radiological reports: an empirical study for adapting the PEGASUS models' family with scarce data
• Convergence and Generalization of Anti-Regularization for Parametric Models
• On Aligning Prediction Models with Clinical Experiential Learning: A Prostate Cancer Case Study
• On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
• Double Descent and Overparameterization in Particle Physics Data
• Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environments
• GRADSTOP: Early Stopping of Gradient Descent via Posterior Sampling
• Models in Decision‐Making Under Risk and Uncertainty
• On Spectral Properties of Gradient-based Explanation Methods
• On Understanding of the Dynamics of Model Capacity in Continual Learning

Related

• Understanding the Double Descent Phenomenon in Deep Learning
• Multi-scale Feature Learning Dynamics: Insights for Double Descent
• When and how epochwise double descent happens
• A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning
• Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle