Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles

2021/03/08 by Gamper, Jevgenij, Rajpoot, Nasir · 5 citations
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2103.05121

Abstract

We present ARCH, a computational pathology (CP) multiple instance captioning dataset to facilitate dense supervision of CP tasks. Existing CP datasets focus on narrow tasks; ARCH on the other hand contains dense diagnostic and morphological descriptions for a range of stains, tissue types and pathologies. Using intrinsic dimensionality estimation, we show that ARCH is the only CP dataset to (ARCH-)rival its computer vision analog MS-COCO Captions. We conjecture that an encoder pre-trained on dense image captions learns transferable representations for most CP tasks. We support the conjecture with evidence that ARCH representation transfers to a variety of pathology sub-tasks better than ImageNet features or representations obtained via self-supervised or multi-task learning on pathology images alone. We release our best model and invite other researchers to test it on their CP tasks.

Cited by

• Cross-Stain Contrastive Learning for Paired Immunohistochemistry and Histopathology Slide Representation Learning
• PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathology
• A Unified Low-level Foundation Model for Enhancing Pathology Image Quality
• HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks
• Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval

Related

• Towards a Visual-Language Foundation Model for Computational Pathology
• Knowledge-enhanced Visual-Language Pretraining for Computational Pathology
• A Deep Learning Approach for Augmenting Perceptional Understanding of Histopathology Images
• PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology
• Inference of captions from histopathological patches