Large-Scale Adversarial Training for Vision-and-Language Representation Learning

2020/06/11 by Zhe Gan, Yen-Chun Chen, Gan, Zhe +9 · 15 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Topic Modeling #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2006.06195

NeurIPS 2020 Spotlight paper

openalex publication_date 2020/06/11 · arxiv created 2020/10/22 · arxiv updated 2020/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.

Citations

• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Adversarial Training Methods for Semi-Supervised Text Classification
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Compositional Attention Networks for Machine Reasoning
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Theoretically Principled Trade-off between Robustness and Accuracy
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Relation-Aware Graph Attention Network for Visual Question Answering
• Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variables
• Are Labels Required for Improving Adversarial Robustness?
• Improving the Robustness of Deep Neural Networks via Adversarial Training with Triplet Loss
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Deep Modular Co-Attention Networks for Visual Question Answering
• Unsupervised Data Augmentation for Consistency Training
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Metric Learning for Adversarial Robustness
• Learning Video Representations using Contrastive Bidirectional Transformer
• Unified Vision-Language Pre-Training for Image Captioning and VQA
• Meta Module Network for Compositional Visual Reasoning
• Adversarial Examples Improve Image Recognition
• Boosting Adversarial Training with Hypersphere Embedding
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Adversarial Training for Large Neural Language Models
• VD-BERT: A Unified Vision and Dialog Transformer with BERT
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Cited by

• Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles
• UniGame: Turning a Unified Multimodal Model Into Its Own Adversary
• Vision-Centric Activation and Coordination for Multimodal Large Language Models
• Towards Self-Refinement of Vision-Language Models with Triangular Consistency
• Improving Generalized Visual Grounding with Instance-aware Joint Learning
• TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
• Referring Expression Instance Retrieval and A Strong End-to-End Baseline
• Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs
• Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities
• Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought
• Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
• SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis
• GeoMM: On Geodesic Perspective for Multi-modal Learning
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
• PATFinger: Prompt-Adapted Transferable Fingerprinting against Unauthorized Multimodal Dataset Usage

Related

• Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
• CAVL: Learning Contrastive and Adaptive Representations of Vision and Language
• Adversarial Prompt Distillation for Vision-Language Models
• AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization
• Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models