Faithful Multimodal Explanation for Visual Question Answering

2018/09/08 by Jialin Wu, Wu, Jialin, Raymond J. Mooney +1 · 5 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1809.02805

openalex publication_date 2018/09/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

AI systems' ability to explain their reasoning is critical to their utility and trustworthiness. Deep neural networks have enabled significant progress on many challenging problems such as visual question answering (VQA). However, most of them are opaque black boxes with limited explanatory capability. This paper presents a novel approach to developing a high-performing VQA system that can elucidate its answers with integrated textual and visual explanations that faithfully reflect important aspects of its underlying reasoning while capturing the style of comprehensible human explanations. Extensive experimental evaluation demonstrates the advantages of this approach compared to competing methods with both automatic evaluation metrics and human evaluation metrics.

Citations

• Least angle regression
• Constituency Parsing with a Self-Attentive Encoder
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• VQA: Visual Question Answering

Cited by

• Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations
• Adversarial Attacks on VQA-NLE: Exposing and Alleviating Inconsistencies in Visual Question Answering Explanations
• Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions
• On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
• On Explaining Visual Captioning with Hybrid Markov Logic Networks

Related

• Multimodal Rationales for Explainable Visual Question Answering
• Interpretable Visual Question Answering Referring to Outside Knowledge
• Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering
• Improving VQA and its Explanations by Comparing Competing Explanations
• Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering