L2 Regularization versus Batch and Weight Normalization

2017/06/16 by Twan van Laarhoven, van Laarhoven, Twan · 211 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Artificial intelligence #Artificial neural network #Computer science #Deep neural networks #Inverse problem #Machine Learning and Algorithms #Machine learning #Mathematical analysis #Mathematics #Normalization (sociology) #Overfitting #Regularization (linguistics) #Regularization perspectives on support vector machines #Stochastic Gradient Optimization Techniques #Tikhonov regularization #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1706.05350

published in arXiv (Cornell University) (Cornell University)

arxiv created 2017/06/16 · openalex publication_date 2017/06/16 · arxiv updated 2017/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.

Cited by

• Scale Weight Decay and Train Better
• Scalable Agentic Reasoning for Designing Biologics Targeting Intrinsically Disordered Proteins
• Three Mechanisms of Weight Decay Regularization
• Towards Understanding Convergence and Generalization of AdamW
• Four Things Everyone Should Know to Improve Batch Normalization
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• Batch Normalization Preconditioning for Neural Network Training
• A statistical theory of cold posteriors in deep neural networks
• Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
• Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• Learned Threshold Pruning
• Weight Decay may matter more than muP for Learning Rate Transfer in Practice
• A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation
• How does the optimizer implicitly bias the model merging loss landscape?
• XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
• Adaptive L2 Regularization in Person Re-Identification
• EmbeddedML: A New Optimized and Fast Machine Learning Library
• On the training dynamics of deep networks with L₂ regularization
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• A Novel Method to Determine Total Oxidant Concentration Produced by Non-Thermal Plasma Based on Image Processing and Machine Learning
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• From Global to Local: A Scalable Benchmark for Local Posterior Sampling
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• How to decay your learning rate
• Recovering Plasticity of Neural Networks via Soft Weight Rescaling
• Fixup Initialization: Residual Learning Without Normalization
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Online Normalization for Training Neural Networks
• On Infinite-Width Hypernetworks
• AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
• Symmetry in Neural Network Parameter Spaces
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• Scaling CrossQ with Weight Normalization
• Optimistic critics can empower small actors
• Shredder: Learning Noise Distributions to Protect Inference Privacy
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Strategies for Language Identification in Code-Mixed Low Resource Languages
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Regularization by Misclassification in ReLU Neural Networks
• A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle
• Improving Learning to Optimize Using Parameter Symmetries

Related

• Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual Networks
• Three Mechanisms of Weight Decay Regularization
• Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalization
• Theoretical Insight into Batch Normalization: Data Dependant Auto-Tuning of Regularization Rate
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks