Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering

2025/06/18 by Brandon Colelough, Colelough, Brandon, Bartels, Davis +2 · 1 citation
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #I.2.7 #Information Retrieval (cs.IR) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.21597

openalex publication_date 2025/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we present an overview of ClinIQLink, a shared task, collocated with the 24th BioNLP workshop at ACL 2025, designed to stress-test large language models (LLMs) on medically-oriented question answering aimed at the level of a General Practitioner. The challenge supplies 4,978 expert-verified, medical source-grounded question-answer pairs that cover seven formats: true/false, multiple choice, unordered list, short answer, short-inverse, multi-hop, and multi-hop-inverse. Participating systems, bundled in Docker or Apptainer images, are executed on the CodaBench platform or the University of Maryland's Zaratan cluster. An automated harness (Task 1) scores closed-ended items by exact match and open-ended items with a three-tier embedding metric. A subsequent physician panel (Task 2) audits the top model responses.

Citations

• Reference-free Evaluation Metrics for Text Generation: A Survey
• The Falcon Series of Open Language Models
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

Cited by

• ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models

Related

• Towards Expert-Level Medical Question Answering with Large Language Models
• Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework
• Enhancing Healthcare through Large Language Models: A Study on Medical Question Answering
• MedLM: Exploring Language Models for Medical Question Answering Systems
• Structured Outputs Enable General-Purpose LLMs to be Medical Experts