Prompt-aligned Gradient for Prompt Tuning

2022/05/30 by Beier Zhu, Yulei Niu, Zhu, Beier +7 · 47 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2205.14865

openalex publication_date 2022/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e.g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]". Therefore, prompt shows a great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the prompt-based similarity measure. However, we find a common failure that improper fine-tuning may not only undermine the prompt's inherent prediction for the task-related classes, but also for other classes in the VLM vocabulary. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompt. We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the the general knowledge learned from VLMs. In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods. Codes are available at https://github.com/BeierZhu/Prompt-align.

Cited by

• Auxiliary Descriptive Knowledge for Few-Shot Adaptation of Vision-Language Model
• Advancing Cache-Based Few-Shot Classification via Patch-Driven Relational Gated Graph Attention
• Decoupling Template Bias in CLIP: Harnessing Empty Prompts for Enhanced Few-Shot Learning
• Dropout Prompt Learning: Towards Robust and Adaptive Vision-Language Models
• From Points to Clouds: Learning Robust Semantic Distributions for Multi-modal Prompts
• VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models
• GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models
• AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learning
• Exploring Weak-to-Strong Generalization for CLIP-based Classification
• ReBaPL: Repulsive Bayesian Prompt Learning
• vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs
• Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models
• FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
• VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuning
• Exploring Cross-Modal Flows for Few-Shot Learning
• VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models
• FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning
• Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptation
• Few-Shot Adaptation Benchmark for Remote Sensing Vision-Language Models
• Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learning
• Referring Expression Comprehension for Small Objects
• VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning
• GeoVLM-R1: Reinforcement Fine-Tuning for Improved Remote Sensing Reasoning
• Alternating Training-based Label Smoothing Enhances Prompt Generalization
• COLA: Context-aware Language-driven Test-time Adaptation
• Informative Text-Image Alignment for Visual Affordance Learning with Foundation Models
• CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization
• Multimodal Representation Learning Conditioned on Semantic Relations
• AttriPrompt: Dynamic Prompt Composition Learning for CLIP
• Causality-guided Prompt Learning for Vision-language Models via Visual Granulation
• Singular Value Few-shot Adaptation of Vision-Language Models
• Resilient Multimodal Industrial Surface Defect Detection with Uncertain Sensors Availability
• Spotlighter: Revisiting Prompt Tuning from a Representative Mining View
• Language-Aware Information Maximization for Transductive Few-Shot CLIP
• Gradient Rectification for Robust Calibration under Distribution Shift
• Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning
• Cross-Domain Few-Shot Learning via Multi-View Collaborative Optimization with Vision-Language Models
• Federated Cross-Modal Style-Aware Prompt Generation
• Transferable Model-agnostic Vision-Language Model Adaptation for Efficient Weak-to-Strong Generalization
• Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
• Generalized Few-Shot Out-of-Distribution Detection
• Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise
• Causal Disentanglement and Cross-Modal Alignment for Enhanced Few-Shot Learning
• Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models
• Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignment
• Decouple before Align: Visual Disentanglement Enhances Prompt Tuning
• MSGCoOp: Multiple Semantic-Guided Context Optimization for Few-Shot Learning

Related

• Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
• Debiased Fine-Tuning for Vision-language Models by Prompt Regularization
• Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
• PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination
• Learning Generalizable Prompt for CLIP with Class Similarity Knowledge