Medical Hallucinations in Foundation Models and Their Impact on Healthcare

2025/02/26 by Yubin Kim, Kim, Yubin, Hyewon Jeong +51 · 12 voices · 27 citations
#cs.CL #cs.AI #cs.CY

paper · pdf · doi:10.48550/arxiv.2503.05777

Abstract

Hallucinations in foundation models arise from autoregressive training objectives that prioritize token-likelihood optimization over epistemic accuracy, fostering overconfidence and poorly calibrated uncertainty. We define medical hallucination as any model-generated output that is factually incorrect, logically inconsistent, or unsupported by authoritative clinical evidence in ways that could alter clinical decisions. We evaluated 11 foundation models (7 general-purpose, 4 medical-specialized) across seven medical hallucination tasks spanning medical reasoning and biomedical information retrieval. General-purpose models achieved significantly higher proportions of hallucination-free responses than medical-specialized models (median: 76.6% vs 51.3%, difference = 25.2%, 95% CI: 18.7-31.3%, Mann-Whitney U = 27.0, p = 0.012, rank-biserial r = -0.64). Top-performing models such as Gemini-2.5 Pro exceeded 97% accuracy when augmented with chain-of-thought prompting (base: 87.6%), while medical-specialized models like MedGemma ranged from 28.6-61.9% despite explicit training on medical corpora. Chain-of-thought reasoning significantly reduced hallucinations in 86.4% of tested comparisons after FDR correction (q < 0.05), demonstrating that explicit reasoning traces enable self-verification and error detection. Physician audits confirmed that 64-72% of residual hallucinations stemmed from causal or temporal reasoning failures rather than knowledge gaps. A global survey of clinicians (n = 70) validated real-world impact: 91.8% had encountered medical hallucinations, and 84.7% considered them capable of causing patient harm. The underperformance of medical-specialized models despite domain training indicates that safety emerges from sophisticated reasoning capabilities and broad knowledge integration developed during large-scale pre-training, not from narrow optimization.

Cited by

• NeuroGRIP: Retrieval-Augmented Graph Refinement for Knowledge-Grounded EEG Seizure Diagnosis
• Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
• Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation
• MedCEG: Reinforcing Verifiable Medical Reasoning with Critical Evidence Graph
• Black-Box Behavioral Distillation Breaks Safety Alignment in Medical LLMs
• Enhancing Lung Cancer Treatment Outcome Prediction through Semantic Feature Engineering Using Large Language Models
• Fine-Tuned LLMs Know They Don't Know: A Parameter-Efficient Approach to Recovering Honesty
• WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue
• Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
• Demo: Statistically Significant Results On Biases and Errors of LLMs Do Not Guarantee Generalizable Results
• Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services
• Disparities in Multilingual LLM-Based Healthcare Q&amp;A
• BioMedSearch: A Multi-Source Biomedical Retrieval Framework Based on LLMs
• An Alternative Trajectory for Generative AI
• Large Language Models Hallucination: A Comprehensive Survey
• Copy-Paste to Mitigate Large Language Model Hallucinations
• Improving Metacognition and Uncertainty Communication in Language Models
• A Field Guide to Deploying AI Agents in Clinical Practice
• CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding
• Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration
• GRPO++: Enhancing Dermatological Reasoning under Low Resource Settings
• Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoning
• Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications
• DualAlign: Generating Clinically Grounded Synthetic Data
• Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate
• Ask ChatGPT: Caveats and Mitigations for Individual Users of AI Chatbots

Discussions

• despite popularised beliefs, LLMs are not fit for medical applications. SoTA models produce "non-trivial levels of hallucinations" even w inference techniques like CoT & search augmented generation: a [bsky, 303 points, 13 comments]
• Really in-depth paper on AI hallucinations in medicine, with lots of discussion and analysis about addressing them & what is appropriate for medicine But I found this bit on how much more accurate th [bsky, 54 points, 2 comments]
• 91% of medical professionals using LLMs have encountered hallucinations and 84% believe they could impact patient health arxiv.org/abs/2503.05777 [bsky, 39 points, 1 comments]
• Cada vez mais estudos do uso de IA em medicina comprovam o que já se sabe de forma empírica: LLM geram sérios riscos às pessoas em ambiente de alto impacto. Este estudo é revelador - LLM alucinam mui [bsky, 9 points, 1 comments]
• Perfect time to post this. arxiv.org/abs/2503.05777 [bsky, 5 points, 0 comments]
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare arxiv.org/abs/2503.05777 #bioinformatics #llms #artificialintelligence [bsky, 3 points, 0 comments]
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [hn, 1 points, 0 comments]
• "Medical Hallucinations in Foundation Models and Their Impact on #Healthcare" arxiv.org/abs/2503.05777 a key limitation of #LLMs is hallucination. This paper examines the unique characteristics, cause [bsky, 1 points, 0 comments]
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare https://arxiv.org/abs/2503.05777 https://github.com/mitmedialab/medical_hallucination [bsky, 1 points, 0 comments]
• O ile propozycje GAI żeby do pizzy dodać klej żeby ciasto się lepiej kleiło uznać możemy za zabawne to halucynacje medyczne są mniej radosne i mogą stanowić zagrożenie dla zdrowia/życia pacjentów arxi [bsky, 0 points, 0 comments]
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare arxiv.org/abs/2503.05777 [bsky, 0 points, 0 comments]
• In a recent paper, researchers performed MedHALT testing on LLMs and surveyed Drs. #Deepseek R1: 90%🤘 for correct answers & 90%🤘 for Halluc. Resistance 👈 #gemini did well too. The most $$$ LLMs & [bsky, 0 points, 1 comments]

Related

• MedHallBench: A New Benchmark for Assessing Hallucination in Medical Large Language Models
• Are Reasoning Models More Prone to Hallucination?
• Trustworthy AI for Medicine: Continuous Hallucination Detection and Elimination with CHECK
• MedHEval: Benchmarking Hallucinations and Mitigation Strategies in Medical Large Vision-Language Models
• Medico: Towards Hallucination Detection and Correction with Multi-source Evidence Fusion