Refiner: Refining Self-attention for Vision Transformers

2021/06/07 by Daquan Zhou, Yujun Shi, Zhou, Daquan +15
Computer Science · Engineering · #Advanced Neural Network Applications #CCD and CMOS Imaging Sensors #Computer Vision and Pattern Recognition (cs.CV) #Currency Recognition and Detection #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2106.03714

openalex publication_date 2021/06/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Vision Transformers (ViTs) have shown competitive accuracy in image classification tasks compared with CNNs. Yet, they generally require much more data for model pre-training. Most of recent works thus are dedicated to designing more complex architectures or training methods to address the data-efficiency issue of ViTs. However, few of them explore improving the self-attention mechanism, a key factor distinguishing ViTs from CNNs. Different from existing works, we introduce a conceptually simple scheme, called refiner, to directly refine the self-attention maps of ViTs. Specifically, refiner explores attention expansion that projects the multi-head attention maps to a higher-dimensional space to promote their diversity. Further, refiner applies convolutions to augment local patterns of the attention maps, which we show is equivalent to a distributed local attention features are aggregated locally with learnable kernels and then globally aggregated with self-attention. Extensive experiments demonstrate that refiner works surprisingly well. Significantly, it enables ViTs to achieve 86% top-1 classification accuracy on ImageNet with only 81M parameters.

Citations

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Fixing the train-test resolution discrepancy
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Talking-Heads Attention
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Language Models are Few-Shot Learners
• Are we done with ImageNet?
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Pre-Trained Image Processing Transformer
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• High-Performance Large-Scale Image Recognition Without Normalization
• DeepViT: Towards Deeper Vision Transformer
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
• LocalViT: Analyzing Locality in Vision Transformers
• Vision Transformers with Patch Diversification

Related

• You Only Need Less Attention at Each Stage in Vision Transformers
• Understanding The Robustness in Vision Transformers
• Skip-Attention: Improving Vision Transformers by Paying Less Attention
• Scratching Visual Transformer's Back with Uniform Attention
• ViT-P: Rethinking Data-efficient Vision Transformers from Locality