Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning

2025/06/21 by Siying Liu, Liu, Shih-Wen, Fan, Hsuan-Yu +6
Computer Science · #AI in cancer detection #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2506.17645

openalex publication_date 2025/06/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Automating medical report generation from histopathology images is a critical challenge requiring effective visual representations and domain-specific knowledge. Inspired by the common practices of human experts, we propose an in-context learning framework called PathGenIC that integrates context derived from the training set with a multimodal in-context learning (ICL) mechanism. Our method dynamically retrieves semantically similar whole slide image (WSI)-report pairs and incorporates adaptive feedback to enhance contextual relevance and generation quality. Evaluated on the HistGen benchmark, the framework achieves state-of-the-art results, with significant improvements across BLEU, METEOR, and ROUGE-L metrics, and demonstrates robustness across diverse report lengths and disease categories. By maximizing training data utility and bridging vision and language with ICL, our work offers a solution for AI-driven histopathology reporting, setting a strong foundation for future advancements in multimodal clinical applications.

Citations

• Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Cross-modal Memory Networks for Radiology Report Generation
• LoRA: Low-Rank Adaptation of Large Language Models
• Generating Radiology Reports via Memory-driven Transformer
• Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
• Language Models are Few-Shot Learners
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Meshed-Memory Transformer for Image Captioning
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Attention Is All You Need

Related

• HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction
• Automatic Report Generation for Histopathology images using pre-trained Vision Transformers
• MPath: Multimodal Pathology Report Generation from Whole Slide Images
• Automatic Report Generation for Histopathology images using pre-trained Vision Transformers and BERT
• Clinical-grade Multi-Organ Pathology Report Generation for Multi-scale Whole Slide Images via a Semantically Guided Medical Text Foundation Model