Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs

2023/05/05 by Somin Wadhwa, Jay DeYoung, Wadhwa, Somin +7 · 9 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Decision Sciences · Engineering · Medicine · Psychology · #Artificial intelligence #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #Computer science #Data extraction #Engineering #FOS: Computer and information sciences #Framing (construction) #MEDLINE #Medicine #Meta-analysis and systematic reviews #Natural language processing #Pathology #Psychological intervention #Psychology #Randomized controlled trial #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.03642

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Results from Randomized Controlled Trials (RCTs) establish the comparative effectiveness of interventions, and are in turn critical inputs for evidence-based care. However, results from RCTs are presented in (often unstructured) natural language articles describing the design, execution, and outcomes of trials; clinicians must manually extract findings pertaining to interventions and outcomes of interest from such articles. This onerous manual process has motivated work on (semi-)automating extraction of structured evidence from trial reports. In this work we propose and evaluate a text-to-text model built on instruction-tuned Large Language Models (LLMs) to jointly extract Interventions, Outcomes, and Comparators (ICO elements) from clinical abstracts, and infer the associated results reported. Manual (expert) and automated evaluations indicate that framing evidence extraction as a conditional generation task and fine-tuning LLMs for this purpose realizes considerable (∼20 point absolute F1 score) gains over the previous SOTA. We perform ablations and error analyses to assess aspects that contribute to model performance, and to highlight potential directions for further improvements. We apply our model to a collection of published RCTs through mid-2022, and release a searchable database of structured findings: http://ico-relations.ebm-nlp.com

Citations

• Transformers: State-of-the-Art Natural Language Processing
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
• Scaling Instruction-Finetuned Language Models

Cited by

• Enhancing Study-Level Inference from Clinical Trial Papers via Reinforcement Learning-Based Numeric Reasoning
• EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers

Related

• Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models
• Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations
• EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes
• Inferring Which Medical Treatments Work from Reports of Clinical Trials
• An automated approach to extracting positive and negative clinical research results