Large-Scale Adversarial Training for Vision-and-Language Representation Learning

2020/06/11 by Gan, Zhe, Chen, Yen-Chun, Li, Linjie +3 · 6 citations
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2006.06195

Abstract

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.

Cited by

• Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles
• UniGame: Turning a Unified Multimodal Model Into Its Own Adversary
• Vision-Centric Activation and Coordination for Multimodal Large Language Models
• Towards Self-Refinement of Vision-Language Models with Triangular Consistency
• Improving Generalized Visual Grounding with Instance-aware Joint Learning
• TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs

Related

• Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
• CAVL: Learning Contrastive and Adaptive Representations of Vision and Language
• Adversarial Prompt Distillation for Vision-Language Models
• AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization
• Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models