Controlling Covariate Shift using Balanced Normalization of Weights

2018/12/11 by Aaron Defazio, Léon Bottou, Defazio, Aaron +1
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Medical Image Segmentation Techniques

paper · pdf · doi:10.48550/arxiv.1812.04549

openalex publication_date 2018/12/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ImageNet classification with deep convolutional neural networks
• Learning Multiple Layers of Features from Tiny Images

Related

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Internal Covariate Shift Bounding Algorithm for Deep Neural Networks by Unitizing Layers' Outputs
• WeightAlign: Normalizing Activations by Weight Alignment
• Delving into Variance Transmission and Normalization: Shift of Average Gradient Makes the Network Collapse
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks