ResMLP: Feedforward networks for image classification with data-efficient training

2021/05/07 by Hugo Touvron, Piotr Bojanowski, Touvron, Hugo +21 · 29 citations
Computer Science · Engineering · Mathematics · Medicine · Neuroscience · #Algorithm #Artificial intelligence #Artificial neural network #Brain Tumor Detection and Classification #COVID-19 diagnosis using AI #Code (set theory) #Computer Vision and Pattern Recognition (cs.CV) #Computer network #Computer science #Contextual image classification #Digital Imaging for Blood Diseases #Engineering #FOS: Computer and information sciences #Feed forward #Image (mathematics) #Independent and identically distributed random variables #Layer (electronics) #Machine learning #Mathematics #Network architecture #Pattern recognition (psychology) #Perceptron #Residual #Set (abstract data type) #Simple (philosophy) #Statistics #Translation (biology) #cs.CV

paper · pdf · open access · doi:10.48550/arxiv.2105.03404

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/05/07 · arxiv created 2021/06/10 · arxiv updated 2021/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) a two-layer feed-forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. Finally, by adapting our model to machine translation we achieve surprisingly good results. We share pre-trained models and our code based on the Timm library.

Cited by

• CertMask: Certifiable Defense Against Adversarial Patches via Theoretically Optimal Mask Coverage
• SAGS: Self-Adaptive Alias-Free Gaussian Splatting for Dynamic Surgical Endoscopic Reconstruction
• UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations
• Simplifying Knowledge Transfer in Pretrained Models
• MoE-Based Learned Inertial Odometry for Bicycle Localization
• Accelerating Dynamic Image Graph Construction on FPGA for Vision GNNs
• CURA: Size Isnt All You Need -- A Compact Universal Architecture for On-Device Intelligence
• A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation
• SV-Mixer: Replacing the Transformer Encoder with Lightweight MLPs for Self-Supervised Model Compression in Speaker Verification
• Chordless cycle filtrations for dimensionality detection in complex networks via topological data analysis
• SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification
• Hierarchical Graph Feature Enhancement with Adaptive Frequency Modulation for Visual Recognition
• Reliable Microservice Tail Latency Prediction via Decoupled Dual-Stream Learning and Gradient Modulation
• Cross-Architecture Distillation Made Simple with Redundancy Suppression
• Self-similarity Analysis in Deep Neural Networks
• Learning Pixel-adaptive Multi-layer Perceptrons for Real-time Image Enhancement
• Combining Transformers and CNNs for Efficient Object Detection in High-Resolution Satellite Imagery
• SpaRTAN: Spatial Reinforcement Token-based Aggregation Network for Visual Recognition
• MobileIE: An Extremely Lightweight and Effective ConvNet for Real-Time Image Enhancement on Mobile Devices
• evMLP: An Efficient Event-Driven MLP Architecture for Vision
• Don't Pay Attention
• Image Recognition with Online Lightweight Vision Transformer: A Survey
• Finger Pose Estimation for Under-screen Fingerprint Sensor
• Expert-Like Reparameterization of Heterogeneous Pyramid Receptive Fields in Efficient CNNs for Fair Medical Image Classification
• Vision Transformers in Precision Agriculture: A Comprehensive Survey
• Optimized Deep Learning for Mammography: Augmentation and Tailored Architectures
• Real-World Depth Recovery via Structure Uncertainty Modeling and Inaccurate GT Depth Fitting
• GFT: Gradient Focal Transformer
• Exploring Synergistic Ensemble Learning: Uniting CNNs, MLP-Mixers, and Vision Transformers to Enhance Image Classification
• Hypergraph Vision Transformers: Images are More than Nodes, More than Edges

Related

• RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
• ResNet strikes back: An improved training procedure in timm
• Multi-level Residual Networks from Dynamical Systems View
• IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
• Study of Residual Networks for Image Recognition