AutoFormer: Searching Transformers for Visual Recognition

2021/07/01 by Minghao Chen, Chen, Minghao, Houwen Peng +5 · 21 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Architecture #Artificial intelligence #Computer science #Domain Adaptation and Few-Shot Learning #Embedding #Engineering #Machine learning #Programming language #Scratch #Transferability #Transformer #Voltage #cs.CV

paper · pdf · doi:10.48550/arxiv.2107.00651

published in arXiv (Cornell University) (Cornell University) · Github: https://github.com/microsoft/AutoML

arxiv created 2021/07/01 · openalex publication_date 2021/07/01 · arxiv updated 2021/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Recently, pure transformer-based models have shown great potentials for vision tasks such as image classification and detection. However, the design of transformer networks is challenging. It has been observed that the depth, embedding dimension, and number of heads can largely affect the performance of vision transformers. Previous models configure these dimensions based upon manual crafting. In this work, we propose a new one-shot architecture search framework, namely AutoFormer, dedicated to vision transformer search. AutoFormer entangles the weights of different blocks in the same layers during supernet training. Benefiting from the strategy, the trained supernet allows thousands of subnets to be very well-trained. Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch. Besides, the searched models, which we refer to AutoFormers, surpass the recent state-of-the-arts such as ViT and DeiT. In particular, AutoFormer-tiny/small/base achieve 74.7%/81.7%/82.4% top-1 accuracy on ImageNet with 5.7M/22.9M/53.7M parameters, respectively. Lastly, we verify the transferability of AutoFormer by providing the performance on downstream benchmarks and distillation experiments. Code and models are available at https://github.com/microsoft/AutoML.

Cited by

• GrowTAS: Progressive Expansion from Small to Large Subnets for Efficient ViT Architecture Search
• Transformers in Vision: A Survey
• CometNet: Contextual Motif-guided Long-term Time Series Forecasting
• Universal Neural Architecture Space: Covering ConvNets, Transformers and Everything in Between
• ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures
• AutoMaAS: Self-Evolving Multi-Agent Architecture Search for Large Language Models
• LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Search
• CoLLM-NAS: Collaborative Large Language Models for Efficient Knowledge-Guided Neural Architecture Search
• Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature
• Vision Generalist Model: A Survey
• Semantic-Enhanced Time-Series Forecasting via Large Language Models
• Tricks and Plug-ins for Gradient Boosting with Transformers
• Compact Vision Transformer by Reduction of Kernel Complexity
• DASViT: Differentiable Architecture Search for Vision Transformer
• NeuTSFlow: Modeling Continuous Functions Behind Time Series Forecasting
• Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
• Bridging the Last Mile of Prediction: Enhancing Time Series Forecasting with Conditional Guided Flow Matching
• S2AFormer: Strip Self-Attention for Efficient Vision Transformer
• AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer
• L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers
• Evolution Meets Diffusion: Efficient Neural Architecture Generation
• Transferrable Surrogates in Expressive Neural Architecture Search Spaces

Related

• AutoTaskFormer: Searching Vision Transformers for Multi-task Learning
• Searching the Search Space of Vision Transformer
• Image Recognition with Online Lightweight Vision Transformer: A Survey
• Three things everyone should know about Vision Transformers
• EfficientFormer: Vision Transformers at MobileNet Speed