LONGQAEVAL: Designing Reliable Evaluations of Long-Form Clinical QA under Resource Constraints

2025/10/12 by Bologna, Federica, Pan, Tiffany, Wilkens, Matthew +2
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.10415

Abstract

Evaluating long-form clinical question answering (QA) systems is resource-intensive and challenging: accurate judgments require medical expertise and achieving consistent human judgments over long-form text is difficult. We introduce LongQAEval, an evaluation framework and set of evaluation recommendations for limited-resource and high-expertise settings. Based on physician annotations of 300 real patient questions answered by physicians and LLMs, we compare coarse answer-level versus fine-grained sentence-level evaluation over the dimensions of correctness, relevance, and safety. We find that inter-annotator agreement (IAA) varies by dimension: fine-grained annotation improves agreement on correctness, coarse improves agreement on relevance, and judgments on safety remain inconsistent. Additionally, annotating only a small subset of sentences can provide reliability comparable to coarse annotations, reducing cost and effort.

Citations

• MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering
• A Scalable Framework for Evaluating Health Language Models
• A Benchmark for Long-Form Medical Question Answering
• Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks
• Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
• K-QA: A Real-World Medical Q&A Benchmark
• Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries
• APPLS: Evaluating Evaluation Metrics for Plain Language Summarization
• LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
• Large Language Models Encode Clinical Knowledge
• Large language models encode clinical knowledge
• Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
• PRAG: Paninian Retrieval-Augmented Generation for Safety-Critical Medical Question Answering
• Inter-Coder Agreement for Computational Linguistics

Related

• A Benchmark for Long-Form Medical Question Answering
• Trustworthy Medical Question Answering: An Evaluation-Centric Survey
• OLAPH: Improving Factuality in Biomedical Long-form Question Answering
• What Would it Take to get Biomedical QA Systems into Practice?
• ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems