Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering

2025/03/14 by Danfeng Guo, Demetri Terzopoulos · 1 voice · 2 citations
Computer Science · #Multimodal Machine Learning Applications

paper · pdf · doi:10.59275/j.melba.2025-1a8b

openalex publication_date 2025/03/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/06/26

Abstract

Large Vision-Language Models (LVLMs) have achieved significant success in recent years, and they have been extended to the medical domain. Although demonstrating satisfactory performance on medical Visual Question Answering (VQA) tasks, Medical LVLMs (MLVLMs) suffer from the hallucination problem, which makes them fail to diagnose complex pathologies. Moreover, they readily fail to learn minority pathologies due to imbalanced training data. We propose two prompting strategies for MLVLMs that reduce hallucination and improve VQA performance. In the first strategy, we provide a detailed explanation of the queried pathology. In the second strategy, we fine-tune a cheap, weak learner to achieve high performance on a specific metric, and textually provide its judgment to the MLVLM. Tested on the MIMIC-CXR-JPG and Chexpert datasets, our methods significantly improve the diagnostic F1 score, with the highest increase being 0.27. We also demonstrate that our prompting strategies can be extended to general LVLM domains. Based on POPE metrics, it effectively suppresses the false negative predictions of existing LVLMs and improves Recall by approximately 0.07.

Cited by

• DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?
• MRGAgents: A Multi-Agent Framework for Improved Medical Report Generation with Med-LVLMs

Discussions

• 🎯Authors propose prompting strategies to mitigate hallucinations and improve diagnostic accuracy in Medical Visual Question Answering using VLMs, by incorporating pathology explanations and weak lear [bsky, 0 points, 0 comments]

Related

• Hallucination Benchmark in Medical Visual Question Answering
• Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
• MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context
• MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Models
• STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering