Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

2020/02/24 by Soham De, De, Soham, Samuel Smith +1 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2002.10444

openalex publication_date 2020/02/24 · openalex created_date 2020/06/25 · openalex updated_date 2026/07/28

Abstract

Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks. We show that this key benefit arises because, at initialization, batch normalization downscales the residual branch relative to the skip connection, by a normalizing factor on the order of the square root of the network depth. This ensures that, early in training, the function computed by normalized residual blocks in deep networks is close to the identity function (on average). We use this insight to develop a simple initialization scheme that can train deep residual networks without normalization. We also provide a detailed empirical study of residual networks, which clarifies that, although batch normalized networks can be trained with larger learning rates, this effect is only beneficial in specific compute regimes, and has minimal benefits when the batch size is small.

Citations

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Wide Residual Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Don't Decay the Learning Rate, Increase the Batch Size
• Three Factors Influencing Minima in SGD
• A Convergence Theory for Deep Learning via Over-Parameterization
• An Empirical Model of Large-Batch Training
• Fixup Initialization: Residual Learning Without Normalization
• The Impact of Neural Network Overparameterization on Gradient Confusion\n and Stochastic Gradient Descent
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• An Exponential Learning Rate Schedule for Deep Learning
• Self-training with Noisy Student improves ImageNet classification
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts

Cited by

• ReZero is All You Need: Fast Convergence at Large Depth
• AlphaGenome: advancing regulatory variant effect prediction with a unified DNA sequence model
• GPS: Harnessing data fusion strategies to improve the accuracy of machine learning-based genomic and phenotypic selection

Related

• Understanding Batch Normalization
• Training Deep Neural Networks Without Batch Normalization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Batch Normalization Explained
• Normalization Propagation: A Parametric Technique for Removing Internal\n Covariate Shift in Deep Networks