Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

2017/02/10 by Sergey Ioffe, Ioffe, Sergey · 8 citations
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1702.03275

openalex publication_date 2017/02/10 · openalex created_date 2017/02/24 · openalex updated_date 2026/07/28

Abstract

Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.

Citations

• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Improved Techniques for Training GANs

Cited by

• The Hidden Power of Normalization Layers in Neural Networks: Exponential Capacity Control
• An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
• Activation Function Design Sustains Plasticity in Continual Learning
• Segmentation and Tracking of Eruptive Solar Phenomena with Convolutional Neural Networks
• Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• E-ConvNeXt: A Lightweight and Efficient ConvNeXt Variant with Cross-Stage Partial Connections
• Generalized few-shot transfer learning architecture for modeling the EDFA gain spectrum

Related

• Theoretical Insight into Batch Normalization: Data Dependant Auto-Tuning of Regularization Rate
• Batchless Normalization: How to Normalize Activations Across Instances with Minimal Memory Requirements
• Decorrelated Batch Normalization
• Four Things Everyone Should Know to Improve Batch Normalization
• Batch normalization does not improve initialization