Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptation

2025/07/01 by Han, Jizhou, Ding, Chenhao, Dong, SongLin +3
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.00462

Abstract

Visual-language models (VLMs) like CLIP exhibit strong generalization but struggle with distribution shifts at test time. Existing training-free test-time adaptation (TTA) methods operate strictly within CLIP's original feature space, relying on high-confidence samples while overlooking the potential of low-confidence ones. We propose MS-TTA, a training-free approach that enhances feature representations beyond CLIP's space using a single-step k-nearest neighbors (kNN) Mean-Shift. By refining all test samples, MS-TTA improves feature compactness and class separability, leading to more stable adaptation. Additionally, a cache of refined embeddings further enhances inference by providing Mean Shift enhanced logits. Extensive evaluations on OOD and cross-dataset benchmarks demonstrate that MS-TTA consistently outperforms state-of-the-art training-free TTA methods, achieving robust adaptation without requiring additional training.

Related

• ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
• Efficient Test-Time Adaptation of Vision-Language Models
• Realistic Test-Time Adaptation of Vision-Language Models
• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
• CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation