Pay Less Attention to Function Words for Free Robustness of Vision-Language Models

2025/12/08 by Tian, Qiwei, Lin, Chenhao, Zhao, Zhengyu +1
Computer Science · #Adversarial Robustness in Machine Learning #Multimodal Machine Learning Applications #Hate Speech and Cyberbullying Detection

paper · doi:10.48550/arxiv.2512.07222

Abstract

To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VLMs against cross-modal adversarial attacks, and propose Function-word De-Attention (FDA) accordingly to mitigate the impact of function words. Similar to differential amplifiers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more aligned and robust VLMs. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and 3 models. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/0.6% performance drops on the 3 tested models on retrieval, and a 90% ASR drop with a 0.3% performance gain on visual grounding. We demonstrate the scalability, generalization, and zero-shot performance of FDA experimentally, as well as in-depth ablation studies and analysis. Code is available at https://github.com/michaeltian108/FDA.

Citations

• Adversarial Video Promotion Against Text-to-Video Retrieval
• EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agent
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
• Qwen3 Technical Report
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
• Qwen2.5-VL Technical Report
• Instant Adversarial Purification with Adversarial Consistency Distillation
• How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM
• Qwen2 Technical Report
• Adversarial Robustness for Visual Grounding of Multimodal Large Language Models
• Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
• Collapse-Aware Triplet Decoupling for Adversarially Robust Image Retrieval
• SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation
• VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models
• Qwen Technical Report
• Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
• Visual Instruction Tuning
• Understanding Zero-Shot Adversarial Robustness for Large-Scale Models
• Towards Adversarial Attack on Vision-Language Pre-training Models
• Detecting Adversarial Perturbations in Multi-Task Perception
• Vision-Language Pre-Training with Triple Contrastive Learning
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Adversarial purification with Score-based generative models
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• BERT-ATTACK: Adversarial Attack Against BERT Using BERT
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• Overfitting in adversarially robust deep learning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Towards Deep Learning Models Resistant to Adversarial Attacks
• On Detecting Adversarial Perturbations
• Modeling Context in Referring Expressions
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for\n Richer Image-to-Sentence Models
• Microsoft COCO: Common Objects in Context
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
• One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
• Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs
• Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models
• Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models