Capabilities of GPT-4 on Medical Challenge Problems

2023/03/20 by Harsha Nori, Nori, Harsha, Nicholas King +7 · 3 voices · 514 citations
Computer Science · Mathematics · Medicine · Psychology · #Artificial Intelligence in Healthcare and Education #Artificial intelligence #Benchmark (surveying) #Calibration #Clinical Reasoning and Diagnostic Skills #Computer science #Counterfactual thinking #Licensure #Machine learning #Mathematics #Medical education #Medical physics #Medical school #Medicine #Psychology #Statistics #Topic Modeling #United States Medical Licensing Examination

paper · pdf · doi:10.48550/arxiv.2303.13375

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/03/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.

Cited by

• Benchmarking Fine-tuning and Retrieval Strategies for a Multimodal Language Model on the NRC Reactor Operator Licensing Examination
• WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics
• LLM-Extracted Covariates for Clinical Causal Inference: Rethinking Integration Strategies
• Safety boundary maintenance in consumer AI systems responding to pediatric health queries: a cross-platform benchmark evaluation under naturalistic and adversarially pressured conditions
• HijackKV: New Threat in Position-Independent KV Cache Reuse
• MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications
• PertReason: A Knowledge-Grounded Benchmark and Framework for Cell-State-Conditioned Mechanistic Reasoning of Perturbation Effects
• Large Language Models as Unified Multimodal Learners for Clinical Prediction
• GraphDx: A Cost-Aware Knowledge-Enhanced Multi-Agent Framework for Sequential Diagnosis
• This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA
• Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
• Sequential Diagnosis with Language Models
• Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare
• Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
• Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs
• The Mirage of LLM Guardrails: A Case Study in AI-Assisted Medical Note Manipulation
• Socioeconomic Inference in LLM Medical Triage: Same Symptoms, Different ZIP Code
• VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs
• HARBOR: Holistic Adaptive Risk assessment model for BehaviORal healthcare
• An Agentic AI Framework for Training General Practitioner Student Skills
• Multi-Agent Medical Decision Consensus Matrix System: An Intelligent Collaborative Framework for Oncology MDT Consultations
• Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM
• COMPARE: Clinical Optimization with Modular Planning and Assessment via RAG-Enhanced AI-OCT: Superior Decision Support for Percutaneous Coronary Intervention Compared to ChatGPT-5 and Junior Operators
• AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
• Multi-Agent Intelligence for Multidisciplinary Decision-Making in Gastrointestinal Oncology
• When Medical AI Explanations Help and When They Harm
• Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework
• CureAgent: A Training-Free Executor-Analyst Framework for Clinical Reasoning
• Automated Identification of Incidentalomas Requiring Follow-Up: A Multi-Anatomy Evaluation of LLM-Based and Supervised Approaches
• MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosis
• AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation
• AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
• Mirror, Mirror on the Wall -- Which is the Best Model of Them All?
• Medical Malice: A Dataset for Context-Aware Safety in Healthcare LLMs
• Assessing Automated Fact-Checking for Medical LLM Responses with Knowledge Graphs
• Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes
• 47B Mixture-of-Experts Beats 671B Dense Models on Chinese Medical Examinations
• Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning
• CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization
• AI as We Describe It: How Large Language Models and Their Applications in Health are Represented Across Channels of Public Discourse
• LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal
• EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis
• SciMON: Scientific Inspiration Machines Optimized for Novelty
• An efficient probabilistic hardware architecture for diffusion-like models
• SALSA: Single-pass Autoregressive LLM Structured Classification
• Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification
• Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
• Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
• Large language models in medicine
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
• Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation
• Open WebUI: An Open, Extensible, and Usable Interface for AI Interaction
• CURE: Confidence-driven Unified Reasoning Ensemble Framework for Medical Question Answering
• Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
• FocusMed: A Large Language Model-based Framework for Enhancing Medical Question Summarization with Focus Identification
• Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks
• Retrieval-Augmented Framework for LLM-Based Clinical Decision Support
• The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability
• Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs
• Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers
• An Improved Framework for Scaling Party Positions from Texts with Transformer
• GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions
• LLM-Based Support for Diabetes Diagnosis: Opportunities, Scenarios, and Challenges with GPT-5
• Position: Human Factors Reshape Adversarial Analysis in Human-AI Decision-Making Systems
• MedLLM: An Open Medical Language Model at the Sub-Billion Scale
• iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
• Beyond Diagnosis: Evaluating Multimodal LLMs for Pathology Localization in Chest Radiographs
• A Knowledge-driven Adaptive Collaboration of LLMs for Enhancing Medical Decision-making
• Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learning
• SitLLM: Large Language Models for Sitting Posture Health Understanding via Pressure Sensor Data
• Towards Better Health Conversations: The Benefits of Context-seeking
• BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
• HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring
• Energy Landscapes Enable Reliable Abstention in Retrieval-Augmented Large Language Models for Healthcare
• RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning
• OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
• Benchmarking GPT-5 for biomedical natural language processing
• Mitigating Clinician Information Overload: Generative AI for Integrated EHR and RPM Data Analysis
• Reasoning LLMs in the Medical Domain: A Literature Survey
• An Investigation on Group Query Hallucination Attacks
• A Comparative Study of Decoding Strategies in Medical Text Generation
• Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncology
• Performance of GPT-5 in Brain Tumor MRI Reasoning
• PASS: Probabilistic Agentic Supernet Sampling for Interpretable and Adaptive Chest X-Ray Reasoning
• AdaptFlow: Adaptive Workflow Optimization via Meta-Learning
• Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need
• Capabilities of GPT-5 on Multimodal Medical Reasoning
• FEAT: A Multi-Agent Forensic AI System with Domain-Adapted Large Language Model for Automated Cause-of-Death Analysis
• On the Limits of Selective AI Prediction: A Case Study in Clinical Decision Making
• Root Cause Analysis Training for Healthcare Professionals With AI-Powered Virtual Simulation: A Proof-of-Concept
• Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction
• Data Overdose? Time for a Quadruple Shot: Knowledge Graph Construction using Enhanced Triple Extraction
• TT-XAI: Trustworthy Clinical Text Explanations via Keyword Distillation and LLM Reasoning
• A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
• Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation
• Enhancing emergency medical service diversion decision-making through large language models integration
• Detecting AI Assistance in Abstract Complex Tasks
• Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
• Extracting epilepsy‐related information from unstructured clinic letters using large language models
• A Language-Driven Framework for Improving Personalized Recommendations: Merging LLMs with Traditional Algorithms
• IyawoBench v2.0: Extended Diagnostic Evaluation of Large Language Model Clinical Triage in Nigerian Primary Care
• DynamiCare: A Dynamic Multi-Agent Framework for Interactive and Open-Ended Medical Decision-Making
• A Clinically-Grounded Two-Stage Framework for Renal CT Report Generation
• MedEthicsQA: A Comprehensive Question Answering Benchmark for Medical Ethics Evaluation of LLMs
• RWESummary: A Framework and Test for Choosing Large Language Models to Summarize Real-World Evidence (RWE) Studies
• A Brain-to-Population Graph Learning Framework for Diagnosing Brain Disorders
• Enhancing Clinical Decision Support and EHR Insights through LLMs and the Model Context Protocol: An Open-Source MCP-FHIR Framework
• LLM-as-a-Fuzzy-Judge: Fine-Tuning Large Language Models as a Clinical Evaluation Judge with Fuzzy Logic
• Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detection
• Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains
• zip2zip: Inference-Time Adaptive Tokenization via Online Compression
• Measuring Sycophancy of Language Models in Multi-turn Dialogues
• Leveraging Interview-Informed LLMs to Model Survey Responses: Comparative Insights from AI-Generated and Human Data
• Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversations
• MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
• MedScore: Generalizable Factuality Evaluation of Free-Form Medical Answers by Domain-adapted Claim Decomposition and Verification
• ChatGPT: Jack of all trades, master of none
• PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language
• Towards medical AI misalignment: a preliminary study
• Collaboration among Multiple Large Language Models for Medical Question Answering
• Revealing Language Model Trajectories via Kullback-Leibler Divergence
• AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation
• Large Language Model Influence on Diagnostic Reasoning
• CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs
• From Questions to Clinical Recommendations: Large Language Models Driving Evidence-Based Clinical Decision Making
• Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection
• HealthBench: Evaluating Large Language Models Towards Improved Human Health
• An integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care
• High-Fidelity Pseudo-label Generation by Large Language Models for Training Robust Radiology Report Classifiers
• Knowledge-Augmented Language Models Interpreting Structured Chest X-Ray Findings
• Gender-Dependent Diagnostic Substitution in LLM Medical Triage: Same Symptoms, Unequal Urgency
• Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
• Talk Before You Retrieve: Agent-Led Discussions for Better RAG in Medical QA
• CuraView: A Multi-Agent Framework for Medical Hallucination Detection with GraphRAG-Enhanced Knowledge Verification
• Multimodal Large Language Models for Medicine: A Comprehensive Survey
• PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering
• Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training
• Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyond
• Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects
• VerifAI: A Verifiable Open-Source Search Engine for Biomedical Question Answering
• A Survey on LLM-Assisted Clinical Trial Recruitment
• Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent
• Evaluating large language models on a highly-specialized topic, radiation oncology physics
• Enhancing systematic reviews in orthodontics: a comparative examination of GPT-3.5 and GPT-4 for generating PICO-based queries with tailored prompts and configurations
• Validity, Reliability, and Transparency in Artificial Intelligence Regulation
• CRAB: A Benchmark for Evaluating Curation of Retrieval-Augmented LLMs in Biomedicine
• Performance of Large Language Models in Supporting Medical Diagnosis and Treatment
• Towards Accurate Differential Diagnosis with Large Language Models
• Assessment of Large Language Models for Enhancing Diabetologist-Developed Personalized Treatment Plans in Complex Type 2 Diabetes
• Foundation model [wikipedia]
• GPT-4 [wikipedia]

Discussions

• Capabilities of GPT-4 on Medical Challenge Problems [hn, 147 points, 135 comments]
• Capabilities of GPT-4 on Medical Challenge Problems [hn, 2 points, 0 comments]
• In our earlier work (arxiv.org/abs/2303.13375) on med benchmarks, we studied potential memorization influences. Recent influences are hard to assess without further study, but we have some evidence th [bsky, 1 points, 1 comments]

Related

• Large Language Models Encode Clinical Knowledge
• ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
• Can large language models reason about medical questions?
• Large language models encode clinical knowledge
• Beyond Overall Accuracy: A Psychometric Deep Dive into the Topic-Specific Medical Capabilities of 80 Large Language Models