Vision Backbone Enhancement via Multi-Stage Cross-Scale Attention

2023/08/10 by Liang Shang, Shang, Liang, Yanli Liu +11 · 1 citation
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Video Surveillance and Tracking Methods #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2308.05872

openalex publication_date 2023/08/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Convolutional neural networks (CNNs) and vision transformers (ViTs) have achieved remarkable success in various vision tasks. However, many architectures do not consider interactions between feature maps from different stages and scales, which may limit their performance. In this work, we propose a simple add-on attention module to overcome these limitations via multi-stage and cross-scale interactions. Specifically, the proposed Multi-Stage Cross-Scale Attention (MSCSA) module takes feature maps from different stages to enable multi-stage interactions and achieves cross-scale interactions by computing self-attention at different scales based on the multi-stage feature maps. Our experiments on several downstream tasks show that MSCSA provides a significant performance boost with modest additional FLOPs and runtime.

Cited by

• SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attention

Related

• CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention
• Cascaded Multi-Scale Attention for Enhanced Multi-Scale Feature Extraction and Interaction with Low-Resolution Images
• Scratching Visual Transformer's Back with Uniform Attention
• You Only Need Less Attention at Each Stage in Vision Transformers
• Multi-grained Attention Networks for Single Image Super-Resolution