Almanac: Retrieval-Augmented Language Models for Clinical Medicine

2023/03/01 by Cyril Zakka, Akash Chaurasia, Zakka, Cyril +14 · 18 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2303.01229

openalex publication_date 2023/03/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n = 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings.

Cited by

• Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoders
• KOM: A Multi-Agent Artificial Intelligence System for Precision Management of Knee Osteoarthritis (KOA)
• Build AI Assistants using Large Language Models and Agents to Enhance the Engineering Education of Biomechanics
• Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
• AI-assisted workflow enables rapid, high-fidelity breast cancer clinical trial eligibility prescreening
• The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability
• CLIN-LLM: A Safety-Constrained Hybrid Framework for Clinical Diagnosis and Treatment Generation
• OncoReason: Structuring Clinical Reasoning in LLMs for Robust and Interpretable Survival Prediction
• Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges
• Exposing Citation Vulnerabilities in Generative Engines
• Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
• Large language models management of medications: three performance analyses
• Transparency of medical artificial intelligence systems
• Cohort-Aware Agents for Individualized Lung Cancer Risk Prediction Using a Retrieval-Augmented Model Selection Framework
• Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
• Exploring Efficiency Frontiers of Thinking Budget in Medical Reasoning: Scaling Laws between Computational Resources and Reasoning Quality
• Reverse Physician-AI Relationship: Full-process Clinical Diagnosis Driven by a Large Language Model
• Fine-Grained Privacy Extraction from Retrieval-Augmented Generation Systems via Knowledge Asymmetry Exploitation

Related

• Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerations
• Towards Evaluating and Building Versatile Large Language Models for Medicine
• Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models
• Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond
• Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report