Robust Calibration of Large Vision-Language Adapters

2024/07/18 by Balamurali Murugesan, Julio Silva-Rodríguez, Murugesan, Balamurali +5 · 14 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Processing Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2407.13588

openalex publication_date 2024/07/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper addresses the critical issue of miscalibration in CLIP-based model adaptation, particularly in the challenging scenario of out-of-distribution (OOD) samples, which has been overlooked in the existing literature on CLIP adaptation. We empirically demonstrate that popular CLIP adaptation approaches, such as Adapters, Prompt Learning, and Test-Time Adaptation, substantially degrade the calibration capabilities of the zero-shot baseline in the presence of distributional drift. We identify the increase in logit ranges as the underlying cause of miscalibration of CLIP adaptation methods, contrasting with previous work on calibrating fully-supervised models. Motivated by these observations, we present a simple and model-agnostic solution to mitigate miscalibration, by scaling the logit range of each sample to its zero-shot prediction logits. We explore three different alternatives to achieve this, which can be either integrated during adaptation or directly used at inference time. Comprehensive experiments on popular OOD classification benchmarks demonstrate the effectiveness of the proposed approaches in mitigating miscalibration while maintaining discriminative performance, whose improvements are consistent across the three families of these increasingly popular approaches. The code is publicly available at: https://github.com/Bala93/CLIPCalib

Cited by

• Calibrating Uncertainty for Zero-Shot Adversarial CLIP
• A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning
• D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Models
• Calibration-Aware Prompt Learning for Medical Vision-Language Models
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey
• Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift
• ViLU: Learning Vision-Language Uncertainties for Failure Prediction
• Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model
• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
• Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction
• PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
• Conformal Prediction for Zero-Shot Models
• Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models

Related

• Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models
• BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptation
• Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
• Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models