Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

2021/06/23 by Qibin Hou, Hou, Qibin, Zihang Jiang +10 · 9 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #cs.CV

paper · pdf · doi:10.48550/arxiv.2106.12368

9 pages

arxiv created 2021/06/23 · openalex publication_date 2021/06/23 · arxiv updated 2021/06/24 · openalex created_date 2021/07/05 · openalex updated_date 2026/07/28

Abstract

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies along one spatial direction and meanwhile preserve precise positional information along the other direction. The resulting position-sensitive outputs are then aggregated in a mutually complementing manner to form expressive representations of the objects of interest. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. Code is available at https://github.com/Andrew-Qibin/VisionPermutator.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ImageNet classification with deep convolutional neural networks
• Wide Residual Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Random Erasing Data Augmentation
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• High-Performance Large-Scale Image Recognition Without Normalization
• DeepViT: Towards Deeper Vision Transformer
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• MLP-Mixer: An all-MLP Architecture for Vision
• Coordinate Attention for Efficient Mobile Network Design

Cited by

• Automated Neural Architecture Design for Industrial Defect Detection
• PointMixer: MLP-Mixer for Point Cloud Understanding
• SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification
• Global Interaction Modelling in Vision Transformer via Super Tokens
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• Learning Pixel-adaptive Multi-layer Perceptrons for Real-time Image Enhancement
• evMLP: An Efficient Event-Driven MLP Architecture for Vision
• THIRDEYE: Cue-Aware Monocular Depth Estimation via Brain-Inspired Multi-Stage Fusion
• CycleMLP: A MLP-like Architecture for Dense Prediction

Related

• MLP-Mixer: An all-MLP Architecture for Vision
• S²-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• MC-MLP:Multiple Coordinate Frames in all-MLP Architecture for Vision
• Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs
• R2-MLP: Round-Roll MLP for Multi-View 3D Object Recognition