HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction

2024/03/08 by Zhengrui Guo, Jiabo Ma, Guo, Zhengrui +9 · 17 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence (cs.AI) #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2403.05396

openalex publication_date 2024/03/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Histopathology serves as the gold standard in cancer diagnosis, with clinical reports being vital in interpreting and understanding this process, guiding cancer treatment and patient care. The automation of histopathology report generation with deep learning stands to significantly enhance clinical efficiency and lessen the labor-intensive, time-consuming burden on pathologists in report writing. In pursuit of this advancement, we introduce HistGen, a multiple instance learning-empowered framework for histopathology report generation together with the first benchmark dataset for evaluation. Inspired by diagnostic and report-writing workflows, HistGen features two delicately designed modules, aiming to boost report generation by aligning whole slide images (WSIs) and diagnostic reports from local and global granularity. To achieve this, a local-global hierarchical encoder is developed for efficient visual feature aggregation from a region-to-slide perspective. Meanwhile, a cross-modal context module is proposed to explicitly facilitate alignment and interaction between distinct modalities, effectively bridging the gap between the extensive visual sequences of WSIs and corresponding highly summarized reports. Experimental results on WSI report generation show the proposed model outperforms state-of-the-art (SOTA) models by a large margin. Moreover, the results of fine-tuning our model on cancer subtyping and survival analysis tasks further demonstrate superior performance compared to SOTA methods, showcasing strong transfer learning capability. Dataset, model weights, and source code are available in https://github.com/dddavid4real/HistGen.

Cited by

• Democratizing Pathology Co-Pilots: An Open Pipeline and Dataset for Whole-Slide Vision-Language Modelling
• LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction
• HARE: an entity and relation centric evaluation framework for histopathology reports
• PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis
• A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generation
• PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generation
• SURE-Med: Systematic Uncertainty Reduction for Enhanced Reliability in Medical Report Generation
• DiagR1: A Vision-Language Model Trained via Reinforcement Learning for Digestive Pathology Diagnosis
• Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation
• LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learning
• CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic
• Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification
• Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining
• Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges
• CheXLearner: Text-Guided Fine-Grained Representation Learning for Progression Detection
• From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology
• CLIP-IT: CLIP-based Pairing for Histology Images Classification

Related

• Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning
• Automatic Report Generation for Histopathology images using pre-trained Vision Transformers
• PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology
• PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction
• ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images