On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?

2024/05/03 by Maxime Zanella, Ismail Ben Ayed, Zanella, Maxime +1 · 11 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2405.02266

openalex publication_date 2024/05/03 · openalex created_date 2024/05/10 · openalex updated_date 2026/07/28

Abstract

The development of large vision-language models, notably CLIP, has catalyzed research into effective adaptation techniques, with a particular focus on soft prompt tuning. Conjointly, test-time augmentation, which utilizes multiple augmented views of a single image to enhance zero-shot generalization, is emerging as a significant area of interest. This has predominantly directed research efforts toward test-time prompt tuning. In contrast, we introduce a robust MeanShift for Test-time Augmentation (MTA), which surpasses prompt-based methods without requiring this intensive training procedure. This positions MTA as an ideal solution for both standalone and API-based applications. Additionally, our method does not rely on ad hoc rules (e.g., confidence threshold) used in some previous test-time augmentation techniques to filter the augmented views. Instead, MTA incorporates a quality assessment variable for each view directly into its optimization process, termed as the inlierness score. This score is jointly optimized with a density mode seeking process, leading to an efficient training- and hyperparameter-free approach. We extensively benchmark our method on 15 datasets and demonstrate MTA's superiority and computational efficiency. Deployed easily as plug-and-play module on top of zero-shot models and state-of-the-art few-shot methods, MTA shows systematic and consistent improvements.

Cited by

• TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models
• From Tokens to Photons: Test-Time Physical Prompting for Vison-Language Models
• T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis
• Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models
• Bayesian Test-time Adaptation for Object Recognition and Detection with Vision-language Models
• EMO-TTA: Improving Test-Time Adaptation of Audio-Language Models for Speech Emotion Recognition
• Training-Free Label Space Alignment for Universal Domain Adaptation
• Language-Aware Information Maximization for Transductive Few-Shot CLIP
• Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey
• ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates

Related

• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
• Noise is an Efficient Learner for Zero-Shot Vision-Language Models
• A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models
• Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models