Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering

2018/12/13 by Peng Gao, Gao Peng, Zhengkai Jiang +13 · 10 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #Multimodal Machine Learning Applications #cs.CV #eess.IV

paper · pdf · doi:10.48550/arxiv.1812.05252

CVPR 2019 ORAL

arxiv created 2019/08/23 · arxiv updated 2019/08/27

Abstract

Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fusing multi-modal features with intra- and inter-modality information flow, which alternatively pass dynamic information between and across the visual and language modalities. It can robustly capture the high-level interactions between language and vision domains, thus significantly improves the performance of visual question answering. We also show that the proposed dynamic intra-modality attention flow conditioned on the other modality can dynamically modulate the intra-modality attention of the target modality, which is vital for multimodality feature fusion. Experimental evaluations on the VQA 2.0 dataset show that the proposed method achieves state-of-the-art VQA performance. Extensive ablation studies are carried out for the comprehensive analysis of the proposed method.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• VQA: Visual Question Answering

Cited by

• DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering
• MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
• CAMA: Enhancing Multimodal In-Context Learning with Context-Aware Modulated Attention
• Multi-modality Latent Interaction Network for Visual Question Answering
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Hadamard product in deep learning: Introduction, Advances and Challenges

Related

• Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering
• Multi-modality Latent Interaction Network for Visual Question Answering
• An Improved Attention for Visual Question Answering
• MGA-VQA: Multi-Granularity Alignment for Visual Question Answering
• Multimodal Unified Attention Networks for Vision-and-Language Interactions