Hire-MLP: Vision MLP via Hierarchical Rearrangement

2021/08/30 by Jianyuan Guo, Yehui Tang, Guo, Jianyuan +13 · 5 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics and Sensor-Based Localization

paper · pdf · doi:10.48550/arxiv.2108.13341

openalex publication_date 2021/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Previous vision MLPs such as MLP-Mixer and ResMLP accept linearly flattened image patches as input, making them inflexible for different input sizes and hard to capture spatial information. Such approach withholds MLPs from getting comparable performance with their transformer-based counterparts and prevents them from becoming a general backbone for computer vision. This paper presents Hire-MLP, a simple yet competitive vision MLP architecture via Hierarchical rearrangement, which contains two levels of rearrangements. Specifically, the inner-region rearrangement is proposed to capture local information inside a spatial region, and the cross-region rearrangement is proposed to enable information communication between different regions and capture global context by circularly shifting all tokens along spatial directions. Extensive experiments demonstrate the effectiveness of Hire-MLP as a versatile backbone for various vision tasks. In particular, Hire-MLP achieves competitive results on image classification, object detection and semantic segmentation tasks, e.g., 83.8% top-1 accuracy on ImageNet, 51.7% box AP and 44.8% mask AP on COCO val2017, and 49.9% mIoU on ADE20K, surpassing previous transformer-based and MLP-based models with better trade-off for accuracy and throughput. Code is available at https://github.com/ggjy/Hire-Wave-MLP.pytorch.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Gradient-based learning applied to document recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Random Erasing Data Augmentation
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• MLP-Mixer: An all-MLP Architecture for Vision
• Global Filter Networks for Image Classification
• PVT v2: Improved baselines with pyramid vision transformer

Cited by

• ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters
• PointMixer: MLP-Mixer for Point Cloud Understanding
• Toward Next-generation Medical Vision Backbones: Modeling Finer-grained Long-range Visual Dependency
• Global Interaction Modelling in Vision Transformer via Super Tokens
• Learning Pixel-adaptive Multi-layer Perceptrons for Real-time Image Enhancement

Related

• S²-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• CS-Mixer: A Cross-Scale Vision MLP Model with Spatial-Channel Mixing
• S²-MLP: Spatial-Shift MLP Architecture for Vision
• Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone