Quilt-1M: One Million Image-Text Pairs for Histopathology

2023/06/20 by Wisdom Oluchi Ikezogwo, Ikezogwo, Wisdom Oluchi, Mehmet Saygın Seyfioğlu +13 · 29 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #AI in cancer detection #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Digital Imaging for Blood Diseases #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2306.11207

openalex publication_date 2023/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has halted comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering 1,087 hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate Quilt: a large-scale vision-language dataset consisting of 768,826 image and text pairs. Quilt was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around 200K samples. We combine Quilt with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: Quilt-1M, with 1M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of Quilt-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across 13 diverse patch-level datasets of 8 different sub-pathologies and cross-modal retrieval tasks.

Citations

• ChatGPT outperforms crowd workers for text-annotation tasks

Cited by

• TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning
• PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
• Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generation
• Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
• Navigating Gigapixel Pathology Images with Large Multimodal Models
• CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection
• Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models
• UniMedVL: Unifying Medical Multimodal Understanding And Generation Through Observation-Knowledge-Analysis
• Hyperparameter Optimization and Reproducibility in Deep Learning Model Training
• PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction
• Visual Instruction Pretraining for Domain-Specific Foundation Models
• Comparing Computational Pathology Foundation Models using Representational Similarity Analysis
• Calibration-Aware Prompt Learning for Medical Vision-Language Models
• Transparency of medical artificial intelligence systems
• PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathology
• Boosting Pathology Foundation Models via Few-shot Prompt-tuning for Rare Cancer Subtyping
• Normal and Abnormal Pathology Knowledge-Augmented Vision-Language Model for Anomaly Detection in Pathology Images
• Controllable Latent Space Augmentation for Digital Pathology
• WISE-FUSE: Efficient Whole Slide Image Encoding via Coarse-to-Fine Patch Selection with VLM and LLM Knowledge Fusion
• HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks
• VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
• Effortless Vision-Language Model Specialization in Histopathology without Annotation
• Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions
• Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translation
• Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning
• MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
• A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics
• IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark
• A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model

Related

• Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos
• ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images
• Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
• Towards a Visual-Language Foundation Model for Computational Pathology
• Boosting Vision-Language Models for Histopathology Classification: Predict all at once