DL³M: A Vision-to-Language Framework for Expert-Level Medical Reasoning through Deep Learning and Large Language Models

2025/12/14 by Md. Hasibul Hasan, Imran Ahmad, Hasan, Md. Najib +11
Computer Science · #Multimodal Machine Learning Applications #Explainable Artificial Intelligence (XAI) #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2512.13742

Abstract

Medical image classifiers detect gastrointestinal diseases well, but they do not explain their decisions. Large language models can generate clinical text, yet they struggle with visual reasoning and often produce unstable or incorrect explanations. This leaves a gap between what a model sees and the type of reasoning a clinician expects. We introduce a framework that links image classification with structured clinical reasoning. A new hybrid model, MobileCoAtNet, is designed for endoscopic images and achieves high accuracy across eight stomach-related classes. Its outputs are then used to drive reasoning by several LLMs. To judge this reasoning, we build two expert-verified benchmarks covering causes, symptoms, treatment, lifestyle, and follow-up care. Thirty-two LLMs are evaluated against these gold standards. Strong classification improves the quality of their explanations, but none of the models reach human-level stability. Even the best LLMs change their reasoning when prompts vary. Our study shows that combining DL with LLMs can produce useful clinical narratives, but current LLMs remain unreliable for high-stakes medical decisions. The framework provides a clearer view of their limits and a path for building safer reasoning systems. The complete source code and datasets used in this study are available at https://github.com/souravbasakshuvo/DL3M.

Citations

• Pitfalls of Evaluating Language Models with Open Benchmarks
• Zero-Shot Multi-Label Classification of Bangla Documents: Large Decoders Vs. Classic Encoders
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Searching for MobileNetV3
• Xception: Deep Learning with Depthwise Separable Convolutions
• Densely Connected Convolutional Networks
• Identity Mappings in Deep Residual Networks
• Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
• Deep Residual Learning for Image Recognition
• Rethinking the Inception Architecture for Computer Vision
• Very Deep Convolutional Networks for Large-Scale Image Recognition

Related

• Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classification
• DiagR1: A Vision-Language Model Trained via Reinforcement Learning for Digestive Pathology Diagnosis
• MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models
• Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models
• DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?