ResNet strikes back: An improved training procedure in timm

2021/10/01 by Ross Wightman, Hugo Touvron, Wightman, Ross +3 · 20 citations
Computer Science · #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2110.00476

openalex publication_date 2021/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture in studies, or as baselines when new architectures are proposed. Yet there has been significant progress on best practices for training neural networks since the inception of the ResNet architecture in 2015. Novel optimization & data-augmentation have increased the effectiveness of the training recipes. In this paper, we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances. We share competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work. For instance, with our more demanding training setting, a vanilla ResNet-50 reaches 80.4% top-1 accuracy at resolution 224x224 on ImageNet-val without extra data or distillation. We also report the performance achieved with popular models with our training procedure.

Citations

• ImageNet classification with deep convolutional neural networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Random Erasing Data Augmentation
• Are we done with ImageNet?
• Learning Multiple Layers of Features from Tiny Images
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• High-Performance Large-Scale Image Recognition Without Normalization
• MLP-Mixer: An all-MLP Architecture for Vision
• Emerging Properties in Self-Supervised Vision Transformers

Cited by

• Galaxy Zoo Evo: 1 million human-annotated images of galaxies
• Advancing Image Classification with Discrete Diffusion Classification Modeling
• Rethinking Vision Transformer Depth via Structural Reparameterization
• BioBench: A Blueprint to Move Beyond ImageNet for Scientific ML Benchmarks
• Neighborhood Feature Pooling for Remote Sensing Image Classification
• Zero-Shot Vehicle Model Recognition via Text-Based Retrieval-Augmented Generation
• Salient Concept-Aware Generative Data Augmentation
• MetaFormer Is Actually What You Need for Vision
• A Fast Knowledge Distillation Framework for Visual Recognition
• Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Adversarial Scheduling
• Unlocking Noise-Resistant Vision: Key Architectural Secrets for Robust Models
• Leveraging Geometric Visual Illusions as Perceptual Inductive Biases for Vision Models
• Robustness of Sensitivity Evaluations for Gravitational Wave Detection Algorithms
• Expandable Residual Approximation for Knowledge Distillation
• Automated Multi-label Classification of Eleven Retinal Diseases: A Benchmark of Modern Architectures and a Meta-Ensemble on a Large Synthetic Dataset
• Beacon: Post-Training Quantization with Integrated Grid Selection
• EventSSEG: Event-driven Self-Supervised Segmentation with Probabilistic Attention
• A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
• Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognition
• DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging

Related

• Revisiting ResNets: Improved Training and Scaling Strategies
• ResNets Are Deeper Than You Think
• Efficient ResNets: Residual Network Design
• ResMLP: Feedforward networks for image classification with data-efficient training
• Multi-level Residual Networks from Dynamical Systems View