A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP

2021/08/30 by Yucheng Zhao, Guangting Wang, Zhao, Yucheng +9 · 3 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2108.13002

openalex publication_date 2021/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Convolutional neural networks (CNN) are the dominant deep neural network (DNN) architecture for computer vision. Recently, Transformer and multi-layer perceptron (MLP)-based models, such as Vision Transformer and MLP-Mixer, started to lead new trends as they showed promising results in the ImageNet classification task. In this paper, we conduct empirical studies on these DNN structures and try to understand their respective pros and cons. To ensure a fair comparison, we first develop a unified framework called SPACH which adopts separate modules for spatial and channel processing. Our experiments under the SPACH framework reveal that all structures can achieve competitive performance at a moderate scale. However, they demonstrate distinctive behaviors when the network size scales up. Based on our findings, we propose two hybrid models using convolution and Transformer modules. The resulting Hybrid-MS-S+ model achieves 83.9% top-1 accuracy with 63M parameters and 12.3G FLOPS. It is already on par with the SOTA models with sophisticated designs. The code and models are publicly available at https://github.com/microsoft/SPACH.

Citations

• ImageNet classification with deep convolutional neural networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Random Erasing Data Augmentation
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• MLP-Mixer: An all-MLP Architecture for Vision
• Global Filter Networks for Image Classification

Cited by

• Selective Masking based Self-Supervised Learning for Image Semantic Segmentation
• DetailSemNet: Elevating Signature Verification through Detail-Semantic Integration
• Hybrid deep learning models for time series forecasting of solar power

Related

• MLP-Mixer: An all-MLP Architecture for Vision
• ImageNet classification with deep convolutional neural networks
• Do Vision Transformers See Like Convolutional Neural Networks?
• Container: Context Aggregation Network
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone