AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation

2025/12/03 by Chuyue Wang, Wang, Chuyue, Jie Feng +9
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Benchmark (surveying) #Blueprint #Bridge (graph theory) #Computation and Language (cs.CL) #Domain (mathematical analysis) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning in Healthcare #Multimodal Machine Learning Applications #Relevance (law) #Scalability #Scheme (mathematics) #Service (business) #Subject-matter expert

paper · pdf · doi:10.48550/arxiv.2512.03737

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/12/03 · openalex created_date 2025/12/05 · openalex updated_date 2026/07/28

Abstract

Accurate and reliable search on online healthcare platforms is critical for user safety and service efficacy. Traditional methods, however, often fail to comprehend complex and nuanced user queries, limiting their effectiveness. Large language models (LLMs) present a promising solution, offering powerful semantic understanding to bridge this gap. Despite their potential, deploying LLMs in this high-stakes domain is fraught with challenges, including factual hallucinations, specialized knowledge gaps, and high operational costs. To overcome these barriers, we introduce AR-Med, a novel framework for Automated Relevance assessment for Medical search that has been successfully deployed at scale on the Online Medical Delivery Platforms. AR-Med grounds LLM reasoning in verified medical knowledge through a retrieval-augmented approach, ensuring high accuracy and reliability. To enable efficient online service, we design a practical knowledge distillation scheme that compresses large teacher models into compact yet powerful student models. We also introduce LocalQSMed, a multi-expert annotated benchmark developed to guide model iteration and ensure strong alignment between offline and online performance. Extensive experiments show AR-Med achieves an offline accuracy of over 93%, a 24% absolute improvement over the original online system, and delivers significant gains in online relevance and user satisfaction. Our work presents a practical and scalable blueprint for developing trustworthy, LLM-powered systems in real-world healthcare applications.

Citations

• R2MED: A Benchmark for Reasoning-Driven Medical Retrieval
• A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval
• AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels
• The Llama 3 Herd of Models
• Don't Make Your LLM an Evaluation Benchmark Cheater
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Capabilities of GPT-4 on Medical Challenge Problems
• Toolformer: Language Models Can Teach Themselves to Use Tools
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• Large Language Models are Zero-Shot Reasoners
• PaLM: Scaling Language Modeling with Pathways
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

Related

• Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models
• Med-R²: Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine
• Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report
• SearchRAG: Can Search Engines Be Helpful for LLM-based Medical Question Answering?
• Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights