ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

2021/06/07 by Yufei Xu, Qiming Zhang, Xu, Yufei +5 · 8 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2106.03348

openalex publication_date 2021/06/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at GitHub.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ImageNet classification with deep convolutional neural networks
• Semantic Understanding of Scenes Through the ADE20K Dataset
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Pre-Trained Image Processing Transformer
• Learning Multiple Layers of Features from Tiny Images
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Conditional Positional Encodings for Vision Transformers
• CvT: Introducing Convolutions to Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
• LocalViT: Analyzing Locality in Vision Transformers
• MLP-Mixer: An all-MLP Architecture for Vision
• Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Cited by

• LitePT: Lighter Yet Stronger Point Transformer
• GrowTAS: Progressive Expansion from Small to Large Subnets for Efficient ViT Architecture Search
• Tab-PET: Graph-Based Positional Encodings for Tabular Transformers
• UniConvNet: Expanding Effective Receptive Field while Maintaining Asymptotically Gaussian Distribution for ConvNets of Any Scale
• VesselRW: Weakly Supervised Subcutaneous Vessel Segmentation via Learned Random Walk Propagation
• DualResolution Residual Architecture with Artifact Suppression for Melanocytic Lesion Segmentation
• Edge Detection for Organ Boundaries via Top Down Refinement and SubPixel Upsampling
• Deeply Dual Supervised learning for melanoma recognition

Related

• ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond
• ECViT: Efficient Convolutional Vision Transformer with Local-Attention and Multi-scale Stages
• How to Train Vision Transformer on Small-scale Datasets?
• IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism
• CvT: Introducing Convolutions to Vision Transformers