A head-to-head comparison of the accuracy of commercially available large language models for infection prevention and control inquiries, 2024

2024/12/12 by Oluchi Abosi, Takaaki Kobayashi, Natalie Ross +4 · 3 voices
Medicine · #Artificial Intelligence in Healthcare and Education

paper · pdf · doi:10.1017/ice.2024.205

Abstract

We investigated the accuracy and completeness of four large language model (LLM) artificial intelligence tools. Most LLMs provided acceptable answers to commonly asked infection prevention questions (accuracy 98.9%, completeness 94.6%). The use of LLMs to supplement infection prevention consults should be further explored.

Discussions

• 🆕 LLM for IP&C Inquiries: Authors investigated four LLM AI tools’ accuracy. Most provided acceptable answers to IP&C questions. Their use to supplement IP&C consults should be further explored. @kb [bsky, 8 points, 1 comments]
• How good are LLMs to answer infection prevention questions? See our recent publication: doi.org/10.1017/ice.... [bsky, 5 points, 1 comments]
• LLMs won’t take our jobs, but learning to work with them is the future. And no, not all LLMs are made equal! In this great collaboration, we tested LLMs for helping infection preventionists—GPT-4 outs [bsky, 3 points, 0 comments]

Related

• Advancing infection prevention and control through artificial intelligence: a scoping review of applications, barriers, and a decision-support checklist
• Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
• Can Large Language Models (LLMs) Act as Virtual Safety Officers?
• Safety challenges of AI in medicine in the era of large language models
• Large Language Models as Agents in the Clinic