C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion

2024/03/21 by Hee Suk Yoon, Yoon, Hee Suk, Eunseop Yoon +9 · 18 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2403.14119

openalex publication_date 2024/03/21 · openalex created_date 2024/03/24 · openalex updated_date 2026/07/28

Abstract

In deep learning, test-time adaptation has gained attention as a method for model fine-tuning without the need for labeled data. A prime exemplification is the recently proposed test-time prompt tuning for large-scale vision-language models such as CLIP. Unfortunately, these prompts have been mainly developed to improve accuracy, overlooking the importance of calibration, which is a crucial aspect for quantifying prediction uncertainty. However, traditional calibration methods rely on substantial amounts of labeled data, making them impractical for test-time scenarios. To this end, this paper explores calibration during test-time prompt tuning by leveraging the inherent properties of CLIP. Through a series of observations, we find that the prompt choice significantly affects the calibration in CLIP, where the prompts leading to higher text feature dispersion result in better-calibrated predictions. Introducing the Average Text Feature Dispersion (ATFD), we establish its relationship with calibration error and present a novel method, Calibrated Test-time Prompt Tuning (C-TPT), for optimizing prompts during test-time with enhanced calibration. Through extensive experiments on different CLIP architectures and datasets, we show that C-TPT can effectively improve the calibration of test-time prompt tuning without needing labeled data. The code is publicly accessible at https://github.com/hee-suk-yoon/C-TPT.

Cited by

• From Tokens to Photons: Test-Time Physical Prompting for Vison-Language Models
• Adapt-As-You-Walk Through the Clouds: Training-Free Online Test-Time Adaptation of 3D Vision-Language Foundation Models
• Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
• Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
• A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models
• When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
• Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models
• D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Models
• TTRV: Test-Time Reinforcement Learning for Vision Language Models
• Bayesian Test-time Adaptation for Object Recognition and Detection with Vision-language Models
• Calibration-Aware Prompt Learning for Medical Vision-Language Models
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
• Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey
• ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
• Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models
• CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text
• Negation-Aware Test-Time Adaptation for Vision-Language Models

Related

• O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models
• Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
• A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models
• CLIPArTT: Adaptation of CLIP to New Domains at Test Time
• A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models