X-Ray-CoT: Interpretable Chest X-ray Diagnosis with Vision-Language Models via Chain-of-Thought Reasoning

2025/08/17 by Chee M. Ng, Ng, Chee, Sun, Liliang +2 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.12455

openalex publication_date 2025/08/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Chest X-ray imaging is crucial for diagnosing pulmonary and cardiac diseases, yet its interpretation demands extensive clinical experience and suffers from inter-observer variability. While deep learning models offer high diagnostic accuracy, their black-box nature hinders clinical adoption in high-stakes medical settings. To address this, we propose X-Ray-CoT (Chest X-Ray Chain-of-Thought), a novel framework leveraging Vision-Language Large Models (LVLMs) for intelligent chest X-ray diagnosis and interpretable report generation. X-Ray-CoT simulates human radiologists' "chain-of-thought" by first extracting multi-modal features and visual concepts, then employing an LLM-based component with a structured Chain-of-Thought prompting strategy to reason and produce detailed natural language diagnostic reports. Evaluated on the CORDA dataset, X-Ray-CoT achieves competitive quantitative performance, with a Balanced Accuracy of 80.52% and F1 score of 78.65% for disease diagnosis, slightly surpassing existing black-box models. Crucially, it uniquely generates high-quality, explainable reports, as validated by preliminary human evaluations. Our ablation studies confirm the integral role of each proposed component, highlighting the necessity of multi-modal fusion and CoT reasoning for robust and transparent medical AI. This work represents a significant step towards trustworthy and clinically actionable AI systems in medical imaging.

Citations

• ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies
• Vision Transformers in Precision Agriculture: A Comprehensive Survey
• Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models
• Few-shot Adaptation of Medical Vision-Language Models
• CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models
• Visual In-Context Learning for Large Vision-Language Models
• Thread of Thought Unraveling Chaotic Contexts
• Interpretable Medical Imagery Diagnosis with Self-Attentive Transformers: A Review of Explainable AI for Health Care
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Explainable Deep Learning Methods in Medical Image Classification: A Survey
• EventBERT: A Pre-Trained Model for Event Correlation Reasoning
• Explainable Artificial Intelligence for Human Decision-Support System in Medical Domain
• Explainable deep learning models in medical image analysis
• A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

Cited by

• OralGPT-Omni: A Versatile Dental Multimodal Large Language Model

Related

• Interpreting Chest X-rays Like a Radiologist: A Benchmark with Clinical Reasoning
• Reasoning Visual Language Model for Chest X-Ray Analysis
• CXR-LanIC: Language-Grounded Interpretable Classifier for Chest X-Ray Diagnosis
• Knowledge-Augmented Language Models Interpreting Structured Chest X-Ray Findings
• CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning