Contextual Evaluation of Large Language Models for Classifying Tropical and Infectious Diseases

2024/09/13 by Mercy Asiedu, Asiedu, Mercy, Nenad Tomašev +21 · 1 citation
Medicine · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Virology and Viral Diseases

paper · pdf · doi:10.48550/arxiv.2409.09201

openalex publication_date 2024/09/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While large language models (LLMs) have shown promise for medical question answering, there is limited work focused on tropical and infectious disease-specific exploration. We build on an opensource tropical and infectious diseases (TRINDs) dataset, expanding it to include demographic and semantic clinical and consumer augmentations yielding 11000+ prompts. We evaluate LLM performance on these, comparing generalist and medical LLMs, as well as LLM outcomes to human experts. We demonstrate through systematic experimentation, the benefit of contextual information such as demographics, location, gender, risk factors for optimal LLM response. Finally we develop a prototype of TRINDs-LM, a research tool that provides a playground to navigate how context impacts LLM outputs for health.

Cited by

• Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information

Related

• Evaluating Large Language Models for Public Health Classification and Extraction Tasks
• Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
• Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases
• Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise
• Combining Domain-Specific Models and LLMs for Automated Disease Phenotyping from Survey Data