FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering

2023/03/19 by Lin, Weizhe, Wang, Zhilin, Byrne, Bill · 2 citations
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2303.10699

Abstract

The widely used Fact-based Visual Question Answering (FVQA) dataset contains visually-grounded questions that require information retrieval using common sense knowledge graphs to answer. It has been observed that the original dataset is highly imbalanced and concentrated on a small portion of its associated knowledge graph. We introduce FVQA 2.0 which contains adversarial variants of test questions to address this imbalance. We show that systems trained with the original FVQA train sets can be vulnerable to adversarial samples and we demonstrate an augmentation scheme to reduce this vulnerability without human annotations.

Cited by

• SEFRQO: A Self-Evolving Fine-Tuned RAG-Based Query Optimizer
• Augmented Vision-Language Models: A Systematic Review

Related

• Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
• FVQA: Fact-Based Visual Question Answering
• FVQA: Fact-based Visual Question Answering
• Human-Adversarial Visual Question Answering
• Adversarial Attacks on VQA-NLE: Exposing and Alleviating Inconsistencies in Visual Question Answering Explanations