A S.C.O.R.E. framework for evaluating open-ended responses from large language models in healthcare

2026/06/25 by Ting Fang Tan, Kabilan Elangovan, JASMINE ONG +12 · 1 voice
Medicine · Health Professions · Computer Science · #Artificial Intelligence in Healthcare and Education #Electronic Health Records Systems #Machine Learning in Healthcare #Consistency (knowledge bases) #Documentation #Reliability (semiconductor) #Health care #Domain (mathematical analysis) #Subject-matter expert #Quality (philosophy) #Internal consistency

paper · doi:10.1016/j.xcrm.2026.102883

openalex publication_date 2026/06/25 · openalex created_date 2026/06/26 · openalex updated_date 2026/07/29

Abstract

Domain-specific evaluation is essential for clinical validation. We propose S.C.O.R.E. (Safety, Consensus & Context, Objectivity, Reproducibility, Explainability), a five-dimensional framework for structured expert evaluation of LLM-generated healthcare responses. S.C.O.R.E. has been validated against quantitative metrics (BLEU, ROUGE, and BERTScore) using three LLMs (GPT-4o, Claude 4 Sonnet, and DeepSeek) across ophthalmology, medication, and anesthesia. While quantitative metrics frequently misclassified clinically appropriate responses as inaccurate, S.C.O.R.E. demonstrated acceptable internal consistency (Cronbach's α 0.745) in the hyperparameter-optimized domain and detected large effect sizes (Cliff's δ 0.68-0.92) reflecting optimization status. Model rankings reversed across specialties-GPT-4o excelled in ophthalmology (optimized domain), while others in non-optimized domains-thus domain-specific tuning is both necessary and detectable through expert evaluation. S.C.O.R.E.'s correlation between framework reliability and optimization status validates its utility for iterative model refinement. This structured approach enables practical clinical validation, providing actionable feedback for developers and supporting regulatory compliance through standardized documentation of safety, evidence alignment, and explainability.

Citations

• LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation
• A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
• Fine-tuning Large Language Model (LLM) Artificial Intelligence Chatbots in Ophthalmology and LLM-based evaluation using GPT-4
• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
• A Survey on Evaluation of Large Language Models
• LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Testing AI on language comprehension tasks reveals insensitivity to underlying meaning
• Large Language Models Encode Clinical Knowledge
• Measuring Massive Multitask Language Understanding
• USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
• PubMedQA: A Dataset for Biomedical Research Question Answering
• BERTScore: Evaluating Text Generation with BERT
• Exploring the Limits of Language Modeling
• A governance model for the application of AI in health care
• Large language models encode clinical knowledge
• Large language models in medicine
• A Survey on Evaluation of Large Language Models
• Testing and Evaluation of Health Care Applications of Large Language Models
• The TRIPOD-LLM reporting guideline for studies using large language models

Discussions

• The S.C.O.R.E. framework - a structured framework for assessing clinical LLM outputs beyond BLEU or ROUGE. Domain-specific evaluation is essential for safe and reliable healthcare AI. #LLM #Healthcare [bsky, 0 points, 0 comments]

Related

• A Proposed S.C.O.R.E. Evaluation Framework for Large Language Models : Safety, Consensus, Objectivity, Reproducibility and Explainability
• Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
• Automatic Evaluation of Healthcare LLMs Beyond Question-Answering
• A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
• LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation