Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation

2025/05/28 by Mehrdad Noori, David Osowiechi, Noori, Mehrdad +15 · 3 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2505.21844

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, test-time adaptation has attracted wide interest in the context of vision-language models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as Open-Vocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VLMs for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. Our approach could be used as plug-and-play for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates a rigorous evaluation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, with a total of 87 distinct test scenarios, establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that our segmentation-tailored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at https://github.com/dosowiechi/MLMP.

Citations

• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• TULIP: Token-length Upgraded CLIP
• FDS: Feedback-guided Domain Synthesis with Multi-Source Conditional Diffusion Models for Domain Generalization
• WATT: Weight Average Test-Time Adaptation of CLIP
• Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
• Long-CLIP: Unlocking the Long-Text Capability of CLIP
• SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference
• Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
• ClusT3: Information Invariant Test-Time Training
• MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation
• On Pitfalls of Test-Time Adaptation
• TFS-ViT: Token-Level Feature Stylization for Domain Generalization
• Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs
• TTTFlow: Unsupervised Test-Time Training with Normalizing Flow
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
• Frozen CLIP Models are Efficient Video Learners
• GroupViT: Semantic Segmentation Emerges from Text Supervision
• Parameter-free Online Test-time Adaptation
• Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
• Extract Free Dense Labels from CLIP
• MEMO: Test Time Robustness via Adaptation and Augmentation
• AudioCLIP: Extending CLIP to Image, Text and Audio
• Source-Free Domain Adaptation for Semantic Segmentation
• Learning Transferable Visual Models From Natural Language Supervision
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• COCO-Stuff: Thing and Stuff Classes in Context
• Playing for Data: Ground Truth from Computer Games
• Cityscapes dataset for semantic urban scene understanding
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Microsoft COCO: Common Objects in Context

Cited by

• Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
• TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
• Purge-Gate: Backpropagation-Free Test-Time Adaptation for Point Clouds Classification via Token Purging

Related

• Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation
• VLMs meet UDA: Boosting Transferability of Open Vocabulary Segmentation with Unsupervised Domain Adaptation
• Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation
• Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
• Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation