Why Do We Need Weight Decay in Modern Deep Learning?

2023/10/06 by Francesco D’Angelo, Francesco D'Angelo, Maksym Andriushchenko +6 · 5 voices · 17 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Multimodal Machine Learning Applications #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.2310.04415

openalex publication_date 2023/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Weight decay is a broadly used technique for training state-of-the-art deep networks from image classification to large language models. Despite its widespread usage and being extensively studied in the classical literature, its role remains poorly understood for deep learning. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For deep networks on vision tasks trained with multipass SGD, we show how weight decay modifies the optimization dynamics enhancing the ever-present implicit regularization of SGD via the loss stabilization mechanism. In contrast, for large language models trained with nearly one-epoch training, we describe how weight decay balances the bias-variance tradeoff in stochastic optimization leading to lower training loss and improved training stability. Overall, we present a unifying perspective from ResNets on vision tasks to LLMs: weight decay is never useful as an explicit regularizer but instead changes the training dynamics in a desirable way. The code is available at https://github.com/tml-epfl/why-weight-decay

Cited by

• Correction of Decoupled Weight Decay
• Tuning for Two Adversaries: Enhancing the Robustness Against Transfer and Query-Based Attacks using Hyperparameter Tuning
• Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Decoder-Only Transformers
• Weight Decay may matter more than muP for Learning Rate Transfer in Practice
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
• Cautious Weight Decay
• Post-Norm can Resharpen Attention
• How does the optimizer implicitly bias the model merging loss landscape?
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• Efficient Hyperparameter Tuning via Trajectory Invariance Principle
• Recovering Plasticity of Neural Networks via Soft Weight Rescaling
• AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
• The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
• Why Gradients Rapidly Increase Near the End of Training
• SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training
• Navigating the Latent Space Dynamics of Neural Models
• Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

Discussions

• "Why Do We Need Weight Decay in Modern Deep Learning?" overparameterized deep networks -> WD changes enhances the implicit regularization of SGD underparameterized models trained with nearly online [bsky, 4 points, 0 comments]
• Why Do We Need Weight Decay in Modern Deep Learning? [hn, 3 points, 0 comments]
• Why do we need weight decay in modern deep learning? [hn, 2 points, 1 comments]
• Why Do We Need Weight Decay in Modern Deep Learning? arxiv.org/abs/2310.04415 [bsky, 1 points, 0 comments]
• Here are the 6 papers that we are presenting at NeurIPS 2024: Main track: 1. Why Do We Need Weight Decay in Modern Deep Learning? (arxiv.org/abs/2310.04415) 2. JailbreakBench: An Open Robustness Benc [bsky, 0 points, 1 comments]

Related

• Understanding Decoupled and Early Weight Decay
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• Three Mechanisms of Weight Decay Regularization
• Approaching Deep Learning through the Spectral Dynamics of Weights
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: ε-shifted L₂ Regularizer