Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering

2020/07/19 by Ruixue Tang, Chao Ma, Tang, Ruixue +7 · 10 citations
Computer Science · #Adversarial system #Artificial intelligence #Code (set theory) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Correctness #Deep learning #Deep neural networks #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Image (mathematics) #Information retrieval #Meaning (existential) #Multimodal Machine Learning Applications #Natural language processing #Programming language #Question answering #Rotation (mathematics) #Source code #cs.CV

paper · pdf · doi:10.48550/arxiv.2007.09592

published in arXiv (Cornell University) (Cornell University) · To appear in ECCV 2020

arxiv created 2020/07/19 · openalex publication_date 2020/07/19 · arxiv updated 2020/07/21 · openalex created_date 2020/07/23 · openalex updated_date 2026/08/05

Abstract

Visual Question Answering (VQA) has achieved great success thanks to the fast development of deep neural networks (DNN). On the other hand, the data augmentation, as one of the major tricks for DNN, has been widely used in many computer vision tasks. However, there are few works studying the data augmentation problem for VQA and none of the existing image based augmentation schemes (such as rotation and flipping) can be directly applied to VQA due to its semantic structure -- an ⟨ image, question, answer⟩ triplet needs to be maintained correctly. For example, a direction related Question-Answer (QA) pair may not be true if the associated image is rotated or flipped. In this paper, instead of directly manipulating images and questions, we use generated adversarial examples for both images and questions as the augmented data. The augmented examples do not change the visual properties presented in the image as well as the semantic meaning of the question, the correctness of the ⟨ image, question, answer⟩ is thus still maintained. We then use adversarial learning to train a classic VQA model (BUTD) with our augmented data. We find that we not only improve the overall performance on VQAv2, but also can withstand adversarial attack effectively, compared to the baseline model. The source code is available at https://github.com/zaynmi/seada-vqa.

Citations

• Adam: A Method for Stochastic Optimization
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Data Noising as Smoothing in Neural Network Language Models
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
• On Evaluating Adversarial Robustness
• Comparing Attention-based Convolutional and Recurrent Neural Networks:\n Success and Limitations in Machine Reading Comprehension
• Adversarial Examples Improve Image Recognition

Cited by

• GenView++: Unifying Adaptive Generative Augmentation and Quality-Driven Supervision for Contrastive Representation Learning
• Semantic Representation Attack against Aligned Large Language Models

Related

• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• Visual Question Answering: A Survey of Methods and Datasets
• Visual Question Answering Using Semantic Information from Image Descriptions
• Visual Question Answering using Deep Learning: A Survey and Performance\n Analysis
• VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models