Effortless Vision-Language Model Specialization in Histopathology without Annotation

2025/08/11 by Jingna Qiu, Qiu, Jingna, Jain, Nishanth +6
Computer Science · #AI in cancer detection #Computer Vision and Pattern Recognition (cs.CV) #Digital Imaging for Blood Diseases #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2508.07835

openalex publication_date 2025/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances in Vision-Language Models (VLMs) in histopathology, such as CONCH and QuiltNet, have demonstrated impressive zero-shot classification capabilities across various tasks. However, their general-purpose design may lead to suboptimal performance in specific downstream applications. While supervised fine-tuning methods address this issue, they require manually labeled samples for adaptation. This paper investigates annotation-free adaptation of VLMs through continued pretraining on domain- and task-relevant image-caption pairs extracted from existing databases. Our experiments on two VLMs, CONCH and QuiltNet, across three downstream tasks reveal that these pairs substantially enhance both zero-shot and few-shot performance. Notably, with larger training sizes, continued pretraining matches the performance of few-shot methods while eliminating manual labeling. Its effectiveness, task-agnostic design, and annotation-free workflow make it a promising pathway for adapting VLMs to new histopathology tasks. Code is available at https://github.com/DeepMicroscopy/Annotation-free-VLM-specialization.

Citations

• Gemma 3 Technical Report
• Low-Rank Few-Shot Adaptation of Vision-Language Models
• Model-based Cleaning of the QUILT-1M Pathology Dataset for Text-Conditional Image Synthesis
• Quilt-1M: One Million Image-Text Pairs for Histopathology
• SuS-X: Training-Free Name-Only Transfer of Vision-Language Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
• PathologyBERT -- Pre-trained Vs. A New Transformer Language Model for Pathology Domain
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Conditional Prompt Learning for Vision-Language Models
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• Learning to Prompt for Vision-Language Models
• Learning Transferable Visual Models From Natural Language Supervision
• A Petri Dish for Histopathology Image Analysis
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach

Related

• Boosting Vision-Language Models for Histopathology Classification: Predict all at once
• Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images
• Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
• Towards a Visual-Language Foundation Model for Computational Pathology
• CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment