WATT: Weight Average Test-Time Adaptation of CLIP

2024/06/19 by David Osowiechi, Osowiechi, David, Mehrdad Noori +17 · 15 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Embedded Systems Design Techniques #FOS: Computer and information sciences #VLSI and Analog Circuit Testing #Video Coding and Compression Technologies

paper · pdf · doi:10.48550/arxiv.2406.13875

openalex publication_date 2024/06/19 · openalex created_date 2024/06/22 · openalex updated_date 2026/07/28

Abstract

Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performance for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a pioneering approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing overall test performance by aggregating diverse textual cues. Our findings underscore the efficacy of WATT in enhancing performance across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without necessitating additional model transformations or trainable modules. Moreover, compared to other Test-Time Adaptation methods, our approach can operate effectively with just a single image. Highlighting the potential of innovative test-time strategies, this research emphasizes their role in fortifying the adaptability of VLMs. The implementation is available at: \urlhttps://github.com/Mehrdad-Noori/WATT.git.

Cited by

• Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts
• Adapt-As-You-Walk Through the Clouds: Training-Free Online Test-Time Adaptation of 3D Vision-Language Foundation Models
• Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions
• TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
• Purge-Gate: Backpropagation-Free Test-Time Adaptation for Point Clouds Classification via Token Purging
• TransForSeg: A Multitask Stereo ViT for Joint Stereo Segmentation and 3D Force Estimation in Catheterization
• Processing and acquisition traces in visual encoders: What does CLIP know about your camera?
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey
• Generalizing vision-language models to novel domains: A comprehensive survey
• AVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time
• Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation
• Ranked Entropy Minimization for Continual Test-Time Adaptation
• Enhancing Monte Carlo Dropout Performance for Uncertainty Quantification
• FRET: Feature Redundancy Elimination for Test Time Adaptation
• Test-time Correlation Alignment

Related

• CLIPArTT: Adaptation of CLIP to New Domains at Test Time
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
• On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?
• ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
• A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models