Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models

2023/03/11 by Will LeVine, LeVine, Will, Benjamin Pikus +5 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2303.12748

openalex publication_date 2023/03/11 · openalex created_date 2023/03/25 · openalex updated_date 2026/07/28

Abstract

Calibration of deep learning models is crucial to their trustworthiness and safe usage, and as such, has been extensively studied in supervised classification models, with methods crafted to decrease miscalibration. However, there has yet to be a comprehensive study of the calibration of vision-language models that are used for zero-shot inference, like CLIP. We measure calibration across relevant variables like prompt, dataset, and architecture, and find that zero-shot inference with CLIP is miscalibrated. Furthermore, we propose a modified version of temperature scaling that is aligned with the common use cases of CLIP as a zero-shot inference model, and show that a single learned temperature generalizes for each specific CLIP model (defined by a chosen pre-training dataset and architecture) across inference dataset and prompt choice.

Cited by

• Calibrating Uncertainty for Zero-Shot Adversarial CLIP
• Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models
• Enabling Fine-Grained Operating Points for Black-Box LLMs

Related

• Robust Calibration of Large Vision-Language Adapters
• An Empirical Study Into What Matters for Calibrating Vision-Language Models
• Open-Vocabulary Calibration for Fine-tuned CLIP
• Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
• BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptation