Adversarial Attacks on VQA-NLE: Exposing and Alleviating Inconsistencies in Visual Question Answering Explanations

2025/08/17 by Yeh, Yahsin, Yi-Lun Wu, Wu, Yilun +4 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.12430

openalex publication_date 2025/08/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Natural language explanations in visual question answering (VQA-NLE) aim to make black-box models more transparent by elucidating their decision-making processes. However, we find that existing VQA-NLE systems can produce inconsistent explanations and reach conclusions without genuinely understanding the underlying context, exposing weaknesses in either their inference pipeline or explanation-generation mechanism. To highlight these vulnerabilities, we not only leverage an existing adversarial strategy to perturb questions but also propose a novel strategy that minimally alters images to induce contradictory or spurious outputs. We further introduce a mitigation method that leverages external knowledge to alleviate these inconsistencies, thereby bolstering model robustness. Extensive evaluations on two standard benchmarks and two widely used VQA-NLE models underscore the effectiveness of our attacks and the potential of knowledge-based defenses, ultimately revealing pressing security and reliability concerns in current VQA-NLE systems.

Citations

• Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
• A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning
• Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
• NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks
• Generated Knowledge Prompting for Commonsense Reasoning
• Towards Transferable Adversarial Attacks on Vision Transformers
• e-ViL: A Dataset and Benchmark for Natural Language Explanations in\n Vision-Language Tasks
• R&R: Metric-guided Adversarial Sentence Generation
• Learning Transferable Visual Models From Natural Language Supervision
• A Self-supervised Approach for Adversarial Robustness
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Transferable Perturbations of Deep Feature Distributions
• BERT-ATTACK: Adversarial Attack Against BERT Using BERT
• Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability
• Counterfactual Samples Synthesizing for Robust Visual Question Answering
• Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets
• Towards Causal VQA: Revealing and Reducing Spurious Correlations by\n Invariant and Covariant Semantic Editing
• Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reduction
• Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
• Enhancing Adversarial Example Transferability with an Intermediate Level Attack
• BERTScore: Evaluating Text Generation with BERT
• Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks
• Cycle-Consistency for Robust Visual Question Answering
• e-SNLI: Natural Language Inference with Natural Language Explanations
• Faithful Multimodal Explanation for Visual Question Answering
• Improving Transferability of Adversarial Examples with Input Diversity
• Multimodal Explanations: Justifying Decisions and Pointing to the\n Evidence
• Towards Deep Learning Models Resistant to Adversarial Attacks
• VQA: Visual Question Answering
• Microsoft COCO: Common Objects in Context

Cited by

• VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search

Related

• Robust Explanations for Visual Question Answering
• Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
• Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
• VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models
• Improving VQA and its Explanations by Comparing Competing Explanations