Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models

2025/08/07 by Xu, Zane, Sun, Jason
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.05237

Abstract

This report synthesizes eight seminal papers on the zero-shot adversarial robustness of vision-language models (VLMs) like CLIP. A central challenge in this domain is the inherent trade-off between enhancing adversarial robustness and preserving the model's zero-shot generalization capabilities. We analyze two primary defense paradigms: Adversarial Fine-Tuning (AFT), which modifies model parameters, and Training-Free/Test-Time Defenses, which preserve them. We trace the evolution from alignment-preserving methods (TeCoA) to embedding space re-engineering (LAAT, TIMA), and from input heuristics (AOM, TTC) to latent-space purification (CLIPure). Finally, we identify key challenges and future directions including hybrid defense strategies and adversarial pre-training.

Citations

• CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP
• CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification
• TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability
• Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
• Learning Transferable Visual Models From Natural Language Supervision

Related

• Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models
• Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
• Understanding Zero-Shot Adversarial Robustness for Large-Scale Models
• One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models