OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

2025/08/29 by Sandhanakrishnan Ravichandran, Ravichandran, Sandhanakrishnan, Shivesh Kumar +15 · 3 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Benchmark (surveying) #Context (archaeology) #Contextual design #Frontier #Machine Learning in Healthcare #Topic Modeling #Trustworthiness #cs.AI #cs.ET #cs.IR #q-bio.QM

paper · pdf · doi:10.48550/arxiv.2509.02594

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/08/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios. Traditional evaluations are often limited to multiple-choice questions that fail to capture essential competencies such as contextual reasoning, contextual awareness, and uncertainty handling. To address these limitations, we evaluate our agentic RAG-based clinical support assistant, DR. INFO, using HealthBench, a rubric-driven benchmark composed of open-ended, expert-annotated health conversations. On the Hard subset of 1,000 challenging examples, DR. INFO achieves a HealthBench Hard score of 0.68, outperforming leading frontier LLMs including the GPT-5 model family (GPT-5: 0.46, GPT-5.2: 0.42, GPT-5.1: 0.40), Grok 3 (0.23), Gemini 2.5 Pro (0.19), and Claude 3.7 Sonnet (0.02) across all behavioral axes (accuracy, completeness, instruction following, etc.). In a separate 100-sample evaluation against similar agentic RAG assistants (OpenEvidence and Pathway.md, now DoxGPT by Doximity), it maintains a performance lead with a HealthBench Hard score of 0.72. These results highlight the strengths of DR. INFO in communication, instruction following, and accuracy, while also revealing areas for improvement in context awareness and response completeness. Overall, the findings underscore the utility of behavior-level, rubric-based evaluation for building reliable and trustworthy AI-enabled clinical support systems.

Citations

• Sequential Diagnosis with Language Models
• HealthBench: Evaluating Large Language Models Towards Improved Human Health
• MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
• Capabilities of GPT-4 on Medical Challenge Problems
• Large Language Models Encode Clinical Knowledge
• Large language models encode clinical knowledge
• What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
• PubMedQA: A Dataset for Biomedical Research Question Answering

Cited by

• DR. INFO at the Point of Care: A Prospective Pilot Study of Physician-Perceived Value of an Agentic Artificial Intelligence Clinical Assistant
• Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages
• MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional

Related

• Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
• PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
• MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
• MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
• MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications