Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models

2025/11/15 by Yunqi Hong, Hong, Yunqi, Kao, Johnson +11
Computer Science · Medicine · #AI in cancer detection #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2511.12008

openalex publication_date 2025/11/15 · openalex created_date 2025/11/19 · openalex updated_date 2026/07/28

Abstract

AI tools in pathology have improved screening throughput, standardized quantification, and revealed prognostic patterns that inform treatment. However, adoption remains limited because most systems still lack the human-readable reasoning needed to audit decisions and prevent errors. We present RECAP-PATH, an interpretable framework that establishes a self-learning paradigm, shifting off-the-shelf multimodal large language models from passive pattern recognition to evidence-linked diagnostic reasoning. At its core is a two-phase learning process that autonomously derives diagnostic criteria: diversification expands pathology-style explanations, while optimization refines them for accuracy. This self-learning approach requires only small labeled sets and no white-box access or weight updates to generate cancer diagnoses. Evaluated on breast and prostate datasets, RECAP-PATH produced rationales aligned with expert assessment and delivered substantial gains in diagnostic accuracy over baselines. By uniting visual understanding with reasoning, RECAP-PATH provides clinically trustworthy AI and demonstrates a generalizable path toward evidence-linked interpretation.

Citations

• Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs
• Gemma 3 Technical Report
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• GPT-4o System Card
• Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Large Language Models for Disease Diagnosis: A Scoping Review
• Characterizing the Interpretability of Attention Maps in Digital Pathology
• Rotary Position Embedding for Vision Transformer
• In-context learning enables multimodal large language models to classify cancer pathology images
• DeepSeek-VL: Towards Real-World Vision-Language Understanding
• Large Language Models as Optimizers
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• How is ChatGPT's behavior changing over time?
• Quilt-1M: One Million Image-Text Pairs for Histopathology
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Large Language Models Are Human-Level Prompt Engineers
• Concept Gradient: Concept-based Interpretation Without Linear Assumption
• BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Images
• Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods
• Transparency of deep neural networks for medical image analysis: A review of interpretability methods
• A Pathology Deep Learning System Capable of Triage of Melanoma Specimens Utilizing Dermatopathologist Consensus as Ground Truth
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images
• Root Mean Square Layer Normalization
• Sanity Checks for Saliency Maps
• Model Cards for Model Reporting
• Representation Learning with Contrastive Predictive Coding
• Attention-based Deep Multiple Instance Learning
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Understanding Black-box Predictions via Influence Functions
• EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers

Related

• Evidence-based diagnostic reasoning with multi-agent copilot for human pathology
• A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model
• DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
• Pathology-CoT: Learning Visual Chain-of-Thought Agent from Expert Whole Slide Image Diagnosis Behavior
• Teaching AI Stepwise Diagnostic Reasoning with Report-Guided Chain-of-Thought Learning