Few-Shot Adversarial Prompt Learning on Vision-Language Models

2024/03/21 by Yiwei Zhou, Zhou, Yiwei, Xiaobo Xia +7 · 10 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Cryptography and Security (cs.CR) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2403.14774

openalex publication_date 2024/03/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The vulnerability of deep neural networks to imperceptible adversarial perturbations has attracted widespread attention. Inspired by the success of vision-language foundation models, previous efforts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. However, in practice, they are still unsatisfactory due to several issues, including heavy adaptation cost, suboptimal text supervision, and uncontrolled natural generalization capacity. In this paper, to address these issues, we propose a few-shot adversarial prompt framework where adapting input sequences with limited data makes significant adversarial robustness improvement. Specifically, we achieve this by providing adversarially correlated text supervision that is end-to-end learned from adversarial examples. We also propose a novel training objective that enhances the consistency of multi-modal features while encourages differentiated uni-modal features between natural and adversarial examples. The proposed framework gives access to learn adversarial text supervision, which provides superior cross-modal adversarial alignment and matches state-of-the-art zero-shot adversarial robustness with only 1% training data. Code is available at: https://github.com/lionel-w2/FAP.

Cited by

• TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models
• Automatic Attack Discovery for Few-Shot Class-Incremental Learning via Large Language Models
• UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation
• Prompt-based Adaptation in Large-scale Vision Models: A Survey
• Provably Robust Adaptation for Language-Empowered Foundation Models
• FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Logic Unseen: Revealing the Logical Blindspots of Vision-Language Models
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• Principled Multimodal Representation Learning

Related

• One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
• Revisiting the Robust Generalization of Adversarial Prompt Tuning
• Understanding Zero-Shot Adversarial Robustness for Large-Scale Models
• Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
• Adversarial Prompt Tuning for Vision-Language Models