Separable Self-attention for Mobile Vision Transformers

2022/06/06 by Sachin Mehta, Mohammad Rastegari, Mehta, Sachin +1 · 23 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Solar Radiation and Photovoltaics

paper · pdf · doi:10.48550/arxiv.2206.02680

openalex publication_date 2022/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires O(k²) time complexity with respect to the number of tokens (or patches) k. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. O(k). A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTv2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTv2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running 3.2× faster on a mobile device. Our source code is available at: \urlhttps://github.com/apple/ml-cvnets

Cited by

• UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
• Towards Test-time Efficient Visual Place Recognition via Asymmetric Query Processing
• Detection of Digital Facial Retouching utilizing Face Beauty Information
• GlimmerNet: A Lightweight Grouped Dilated Depthwise Convolutions for UAV-Based Emergency Monitoring
• Transformer Driven Visual Servoing for Fabric Texture Matching Using Dual-Arm Manipulator
• CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer
• AdaptViG: Adaptive Vision GNN with Exponential Decay Gating
• How Do Data Owners Say No? A Case Study of Data Consent Mechanisms in Web-Scraped Vision-Language AI Training Datasets
• EuroMineNet: A Multitemporal Sentinel-2 Benchmark for Spatiotemporal Mining Footprint Analysis in the European Union (2015-2024)
• MetaChest: Generalized few-shot learning of pathologies from chest X-rays
• EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View
• TFD-former: Time-frequency domain fusion decoders for effective and robust fault diagnosis under time-varying speeds
• CoAtNeXt:An Attention-Enhanced ConvNeXtV2-Transformer Hybrid Model for Gastric Tissue Classification
• VCMamba: Bridging Convolutions with Multi-Directional Mamba for Efficient Visual Representation
• CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference
• ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers
• Mobile-Friendly Deep Learning for Plant Disease Detection: A Lightweight CNN Benchmark Across 101 Classes of 33 Crops
• UniConvNet: Expanding Effective Receptive Field while Maintaining Asymptotically Gaussian Distribution for ConvNets of Any Scale
• Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms
• Analysis of Hyperparameter Optimization Effects on Lightweight Deep Models for Real-Time Image Classification
• Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models
• DPFA-net: a lightweight hybrid neural network with dual path feature aggregation for food image recognition
• LAID: Lightweight AI-Generated Image Detection in Spatial and Spectral Domains

Related

• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications
• MoCoViT: Mobile Convolutional Vision Transformer
• SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design
• You Only Need Less Attention at Each Stage in Vision Transformers