2024/10/24 by Mohit Chandra, Chandra, Mohit, Gaurav Verma +12 · 6 citations
Health Professions · Medicine · Pharmacology, Toxicology and Pharmaceutics · #Artificial Intelligence (cs.AI) #Biomedical Ethics and Regulation #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Medical Malpractice and Liability Issues #Pharmaceutical industry and healthcare
paper · pdf · doi:10.48550/arxiv.2410.19155
openalex publication_date 2024/10/24 · openalex created_date 2024/11/14 · openalex updated_date 2026/07/28
Adverse Drug Reactions (ADRs) from psychiatric medications are the leading cause of hospitalizations among mental health patients. With healthcare systems and online communities facing limitations in resolving ADR-related issues, Large Language Models (LLMs) have the potential to fill this gap. Despite the increasing capabilities of LLMs, past research has not explored their capabilities in detecting ADRs related to psychiatric medications or in providing effective harm reduction strategies. To address this, we introduce the Psych-ADR benchmark and the Adverse Drug Reaction Response Assessment (ADRA) framework to systematically evaluate LLM performance in detecting ADR expressions and delivering expert-aligned mitigation strategies. Our analyses show that LLMs struggle with understanding the nuances of ADRs and differentiating between types of ADRs. While LLMs align with experts in terms of expressed emotions and tone of the text, their responses are more complex, harder to read, and only 70.86% aligned with expert strategies. Furthermore, they provide less actionable advice by a margin of 12.32% on average. Our work provides a comprehensive benchmark and evaluation framework for assessing LLMs in strategy-driven tasks within high-risk domains.