2024/10/11 by Yu He Ke, Liyuan Jin, Ke, Yu He +23
Computer Science · Health Professions · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Online Learning and Analytics
paper · pdf · doi:10.48550/arxiv.2410.08431
openalex publication_date 2024/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Large Language Models (LLMs) show potential for medical applications but often lack specialized clinical knowledge. Retrieval Augmented Generation (RAG) allows customization with domain-specific information, making it suitable for healthcare. This study evaluates the accuracy, consistency, and safety of RAG models in determining fitness for surgery and providing preoperative instructions. We developed LLM-RAG models using 35 local and 23 international preoperative guidelines and tested them against human-generated responses. A total of 3,682 responses were evaluated. Clinical documents were processed using Llamaindex, and 10 LLMs, including GPT3.5, GPT4, and Claude-3, were assessed. Fourteen clinical scenarios were analyzed, focusing on seven aspects of preoperative instructions. Established guidelines and expert judgment were used to determine correct responses, with human-generated answers serving as comparisons. The LLM-RAG models generated responses within 20 seconds, significantly faster than clinicians (10 minutes). The GPT4 LLM-RAG model achieved the highest accuracy (96.4% vs. 86.6%, p=0.016), with no hallucinations and producing correct instructions comparable to clinicians. Results were consistent across both local and international guidelines. This study demonstrates the potential of LLM-RAG models for preoperative healthcare tasks, highlighting their efficiency, scalability, and reliability.