X-Linear Attention Networks for Image Captioning

2020/03/31 by Yingwei Pan, Ting Yao, Pan, Yingwei +5 · 13 citations
Computer Science · Mathematics · #Advanced Image and Video Retrieval Techniques #Algorithm #Artificial intelligence #Bilinear interpolation #Block (permutation group theory) #Closed captioning #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Domain Adaptation and Few-Shot Learning #Encoder #FOS: Computer and information sciences #Feature (linguistics) #Image (mathematics) #Leverage (statistics) #Mathematics #Multimodal Machine Learning Applications #Pooling #Sentence #Theoretical computer science #cs.CV

paper · pdf · doi:10.48550/arxiv.2003.14080

published in arXiv (Cornell University) (Cornell University) · CVPR 2020; The source code and model are publicly available at: https://github.com/Panda-Peter/image-captioning

arxiv created 2020/03/31 · openalex publication_date 2020/03/31 · arxiv updated 2020/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent progress on fine-grained visual recognition and visual question answering has featured Bilinear Pooling, which effectively models the 2^nd order interactions across multi-modal inputs. Nevertheless, there has not been evidence in support of building such interactions concurrently with attention mechanism for image captioning. In this paper, we introduce a unified attention block -- X-Linear attention block, that fully employs bilinear pooling to selectively capitalize on visual information or perform multi-modal reasoning. Technically, X-Linear attention block simultaneously exploits both the spatial and channel-wise bilinear attention distributions to capture the 2^nd order interactions between the input single-modal or multi-modal features. Higher and even infinity order feature interactions are readily modeled through stacking multiple X-Linear attention blocks and equipping the block with Exponential Linear Unit (ELU) in a parameter-free fashion, respectively. Furthermore, we present X-Linear Attention Networks (dubbed as X-LAN) that novelly integrates X-Linear attention block(s) into image encoder and sentence decoder of image captioning model to leverage higher order intra- and inter-modal interactions. The experiments on COCO benchmark demonstrate that our X-LAN obtains to-date the best published CIDEr performance of 132.0% on COCO Karpathy test split. When further endowing Transformer with X-Linear attention blocks, CIDEr is boosted up to 132.8%. Source code is available at \urlhttps://github.com/Panda-Peter/image-captioning.

Citations

• Sequence to Sequence Learning with Neural Networks
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Continuously Differentiable Exponential Linear Units
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Image Captioning with Semantic Attention

Cited by

• Diffusion Is Your Friend in Show, Suggest and Tell
• Fast SceneScript: Accurate and Efficient Structured Language Model via Multi-Token Prediction
• SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning
• DescribeEarth: Describe Anything for Remote Sensing Images
• RORPCap: Retrieval-based Objects and Relations Prompt for Image Captioning
• AGIC: Attention-Guided Image Captioning to Improve Caption Relevance
• On Explaining Visual Captioning with Hybrid Markov Logic Networks
• Learnable Retrieval Enhanced Visual-Text Alignment and Fusion for Radiology Report Generation
• Efficiency Robustness of Dynamic Deep Learning Systems
• Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation
• SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks
• MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion
• DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation

Related

• Bilinear Attention Networks
• Areas of Attention for Image Captioning
• Gated Hierarchical Attention for Image Captioning
• Guiding Attention using Partial-Order Relationships for Image Captioning
• Attention on Attention for Image Captioning