High-Performance Large-Scale Image Recognition Without Normalization

2021/02/11 by Andrew Brock, Brock, Andrew, Soham De +6 · 2 voices · 35 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2102.06171

openalex publication_date 2021/02/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

Citations

• On the difficulty of training Recurrent Neural Networks
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Stochastic Approximation Method
• ImageNet classification with deep convolutional neural networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Self-Attention Generative Adversarial Networks
• Four Things Everyone Should Know to Improve Batch Normalization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Scaling Laws for Neural Language Models
• Self-Training With Noisy Student Improves ImageNet Classification
• Momentum Contrast for Unsupervised Visual Representation Learning
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Array programming with NumPy
• LambdaNetworks: Modeling Long-Range Interactions Without Attention

Cited by

• AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
• Dataset Distillation by Influence Matching
• Sharp Stability Threshold and Certification for Designing Stable Residual Architectures
• PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark
• Transformers without Normalization
• Stronger Normalization-Free Transformers
• Disentangled and Distilled Encoder for Out-of-Distribution Reasoning with Rademacher Guarantees
• Breast Cell Segmentation Under Extreme Data Constraints: Quantum Enhancement Meets Adaptive Loss Stabilization
• Private Zeroth-Order Optimization with Public Data
• Estimating Bidirectional Causal Effects with Large Scale Online Kernel Learning
• ResNet strikes back: An improved training procedure in timm
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder
• Generalisation of automatic tumour segmentation in histopathological whole-slide images across multiple cancer types
• Closed-Form Last Layer Optimization
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Hybrid Dual-Batch and Cyclic Progressive Learning for Efficient Distributed Training
• NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
• Efficient Multimodal Dataset Distillation via Generative Models
• Rethinking "Batch" in BatchNorm
• Revisiting ResNets: Improved Training and Scaling Strategies
• XCiT: Cross-Covariance Image Transformers
• STA-Net: A Decoupled Shape and Texture Attention Network for Lightweight Plant Disease Classification
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• Investigating Transfer Learning Capabilities of Vision Transformers and CNNs by Fine-Tuning a Single Trainable Block
• A Self-Ensemble Inspired Approach for Effective Training of Binary-Weight Spiking Neural Networks
• PEdger++: Practical Edge Detection via Assembling Cross Information
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• E-CaTCH: Event-Centric Cross-Modal Attention with Temporal Consistency and Class-Imbalance Handling for Misinformation Detection
• Forecasting West Nile virus with deep graph encoders
• Normalization (machine learning) [wikipedia]
• Vision-language model [wikipedia]
• Weight initialization [wikipedia]

Discussions

• DeepMind achieves SOTA image recognition with 8.7x faster training [hn, 291 points, 83 comments]
• High-Performance Large-Scale Image Recognition Without Normalization [hn, 11 points, 3 comments]

Related

• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Image Classification at Supercomputer Scale
• Batch Group Normalization
• Accelerating Training of Deep Neural Networks with a Standardization Loss