Human-in-the-Loop Large Language Model Framework for Identification of Cutaneous Immune-Related Adverse Events

2026/05/09 by Charles Lu, Olivia Burke, Debby Cheng +14
#cs.CL #cs.HC #cs.MA

paper · pdf

Abstract

This study evaluated a retrieval-augmented, multi-agent large language model (LLM)-driven, human-in-the-loop framework for detecting cutaneous immune-related adverse events (cirAEs) from clinical notes. Compared with unassisted manual review, the LLM-assisted workflow improved accuracy (F1 = 0.88 vs 0.77), inter-rater agreement measured by Cohen's kappa (kappa = 0.82 vs 0.50), and reduced average review time by approximately half. This framework pilots how LLMs can be applied to identify immune-related toxicities across organ systems and, more broadly, enable accurate, scalable, and transparent adverse event data extraction.

Related

• Performance of a large language model for identifying central line-associated bloodstream infections (CLABSI) using real clinical notes
• Large language models accurately identify immunosuppression in intensive care unit patients
• Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events
• Institutional-Level Monitoring of Immune Checkpoint Inhibitor IrAEs Using a Novel Natural Language Processing Algorithmic Pipeline
• Improving Entity Recognition Using Ensembles of Deep Learning and Fine-tuned Large Language Models: A Case Study on Adverse Event Extraction from Multiple Sources