Locally Enhanced Self-Attention: Combining Self-Attention and Convolution as Local and Context Terms

2021/07/12 by Chenglin Yang, Yang, Chenglin, Siyuan Qiao +5
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2107.05637

openalex publication_date 2021/07/12 · openalex created_date 2021/12/06 · openalex updated_date 2026/07/28

Abstract

Self-Attention has become prevalent in computer vision models. Inspired by fully connected Conditional Random Fields (CRFs), we decompose self-attention into local and context terms. They correspond to the unary and binary terms in CRF and are implemented by attention mechanisms with projection matrices. We observe that the unary terms only make small contributions to the outputs, and meanwhile standard CNNs that rely solely on the unary terms achieve great performances on a variety of tasks. Therefore, we propose Locally Enhanced Self-Attention (LESA), which enhances the unary term by incorporating it with convolutions, and utilizes a fusion module to dynamically couple the unary and binary operations. In our experiments, we replace the self-attention modules with LESA. The results on ImageNet and COCO show the superiority of LESA over convolution and self-attention baselines for the tasks of image recognition, object detection, and instance segmentation. The code is made publicly available.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Long Short-Term Memory
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet classification with deep convolutional neural networks
• Wide Residual Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Scaling Wide Residual Networks for Panoptic Segmentation
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Conditional Positional Encodings for Vision Transformers
• CvT: Introducing Convolutions to Vision Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• PVT v2: Improved baselines with pyramid vision transformer

Related

• Stand-Alone Self-Attention in Vision Models
• Attention Augmented Convolutional Networks
• Scaling Local Self-Attention for Parameter Efficient Visual Backbones
• EAANet: Efficient Attention Augmented Convolutional Networks
• Exploring Self-attention for Image Recognition