2021/06/04 by Xuanhong Chen, Hang Wang, Chen, Xuanhong +3
Computer Science · #Advanced Image and Video Retrieval Techniques #Computability, Logic, AI Algorithms #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Topological and Geometric Data Analysis
paper · pdf · doi:10.48550/arxiv.2106.02253
openalex publication_date 2021/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Convolution and self-attention are acting as two fundamental building blocks in deep neural networks, where the former extracts local image features in a linear way while the latter non-locally encodes high-order contextual relationships. Though essentially complementary to each other, i.e., first-/high-order, stat-of-the-art architectures, i.e., CNNs or transformers lack a principled way to simultaneously apply both operations in a single computational module, due to their heterogeneous computing pattern and excessive burden of global dot-product for visual tasks. In this work, we theoretically derive a global self-attention approximation scheme, which approximates a self-attention via the convolution operation on transformed features. Based on the approximated scheme, we establish a multi-branch elementary module composed of both convolution and self-attention operation, capable of unifying both local and non-local feature interaction. Importantly, once trained, this multi-branch module could be conditionally converted into a single standard convolution operation via structural re-parameterization, rendering a pure convolution styled operator named X-volution, ready to be plugged into any modern networks as an atomic operation. Extensive experiments demonstrate that the proposed X-volution, achieves highly competitive visual understanding improvements (+1.2% top-1 accuracy on ImageNet classification, +1.7 box AP and +1.5 mask AP on COCO detection and segmentation).