Clinical knowledge in LLMs does not translate to human interactions

2025/04/26 by Andrew M. Bean, Bean, Andrew M., Rebecca Payne +19 · 15 voices · 5 citations
Medicine · Computer Science · #Artificial Intelligence in Healthcare and Education #Global Health and Surgery #Explainable Artificial Intelligence (XAI)

paper · pdf · doi:10.48550/arxiv.2504.18919

Abstract

Global healthcare providers are exploring use of large language models (LLMs) to provide medical advice to the public. LLMs now achieve nearly perfect scores on medical licensing exams, but this does not necessarily translate to accurate performance in real-world settings. We tested if LLMs can assist members of the public in identifying underlying conditions and choosing a course of action (disposition) in ten medical scenarios in a controlled study with 1,298 participants. Participants were randomly assigned to receive assistance from an LLM (GPT-4o, Llama 3, Command R+) or a source of their choice (control). Tested alone, LLMs complete the scenarios accurately, correctly identifying conditions in 94.9% of cases and disposition in 56.3% on average. However, participants using the same LLMs identified relevant conditions in less than 34.5% of cases and disposition in less than 44.2%, both no better than the control group. We identify user interactions as a challenge to the deployment of LLMs for medical advice. Standard benchmarks for medical knowledge and simulated patient interactions do not predict the failures we find with human participants. Moving forward, we recommend systematic human user testing to evaluate interactive capabilities prior to public deployments in healthcare.

Cited by

• Can LLMs Understand What We Cannot Say? Measuring Multilevel Alignment Through Abortion Stigma Across Cognitive, Interpersonal, and Structural Levels
• Generalist Foundation Models Are Not Clinical Enough for Hospital Operations
• A Locally Executable AI System for Improving Preoperative Patient Communication: A Multi-Domain Clinical Evaluation
• Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries
• Towards Better Health Conversations: The Benefits of Context-seeking

Discussions

• Clinical knowledge in LLMs does not translate to human interactions [hn, 102 points, 39 comments]
• Large language models (LLMs) are now reaching the level of doctors in medical benchmarks. But they fail when they are supposed to communicate with people. There they are hardly more helpful than searc [bsky, 7 points, 0 comments]
• 🤖The amount of trust placed in these models is especially worrying because we still don’t know how good they are at helping people manage their health. ⚠️A study by researchers at Oxford, which has b [bsky, 4 points, 0 comments]
• Large language models aren’t coming for my job quite yet… preprint of an exciting new paper led by my Oxford Internet colleagues arxiv.org/abs/2504.18919 [bsky, 2 points, 0 comments]
• The ability of LLMs to pass medical exams is much touted. Unfortunately, patients don't frame their description of symptoms in the way textbooks describe, which makes LLMs very bad at connecting sympt [bsky, 1 points, 0 comments]
• Direct link to the paper: arxiv.org/pdf/2504.18919 [bsky, 1 points, 0 comments]
• Clinical knowledge in LLMs does not translate to human interactions #HackerNews https://arxiv.org/pdf/2504.18919 [bsky, 0 points, 0 comments]
• Clinical knowledge in LLMs does not translate to human interactions https://arxiv.org/pdf/2504.18919 [bsky, 0 points, 0 comments]
• Clinical knowledge in LLMs does not translate to human interactions https://arxiv.org/pdf/2504.18919 [bsky, 0 points, 0 comments]
• Clinical knowledge in LLMs does not translate to human interactions https://arxiv.org/pdf/2504.18919 (https://news.ycombinator.com/item?id=44279209) [bsky, 0 points, 0 comments]
• Clinical knowledge in LLMs does not translate to human interactions https://arxiv.org/pdf/2504.18919 (https://news.ycombinator.com/item?id=44279209) [bsky, 0 points, 0 comments]
• https://bsky.app/profile/buzzing.cc.web.brid.gy/post/3lrq2aegasou2 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2504.18919 [bsky, 0 points, 0 comments]
• Clinical knowledge in LLMs does not translate to human interactions [Discussion] [bsky, 0 points, 0 comments]
• Clinical knowledge in LLMs does not translate to human interactions [bsky, 0 points, 0 comments]

Related

• Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
• Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domain
• Beyond Multiple-Choice Accuracy: Real-World Challenges of Implementing Large Language Models in Healthcare
• Large Language Models as Agents in the Clinic
• Enhancing Patient-Centric Communication: Leveraging LLMs to Simulate Patient Perspectives