Are Smaller Open-Weight LLMs Closing the Gap to Proprietary Models for Biomedical Question Answering?

2025/09/23 by Damian Stachura, Stachura, Damian, Joanna Konieczna +3
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.18843

openalex publication_date 2025/09/23 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

Open-weight versions of large language models (LLMs) are rapidly advancing, with state-of-the-art models like DeepSeek-V3 now performing comparably to proprietary LLMs. This progression raises the question of whether small open-weight LLMs are capable of effectively replacing larger closed-source models. We are particularly interested in the context of biomedical question-answering, a domain we explored by participating in Task 13B Phase B of the BioASQ challenge. In this work, we compare several open-weight models against top-performing systems such as GPT-4o, GPT-4.1, Claude 3.5 Sonnet, and Claude 3.7 Sonnet. To enhance question answering capabilities, we use various techniques including retrieving the most relevant snippets based on embedding distance, in-context learning, and structured outputs. For certain submissions, we utilize ensemble approaches to leverage the diverse outputs generated by different models for exact-answer questions. Our results demonstrate that open-weight LLMs are comparable to proprietary ones. In some instances, open-weight LLMs even surpassed their closed counterparts, particularly when ensembling strategies were applied. All code is publicly available at https://github.com/evidenceprime/BioASQ-13b.

Citations

• Overview of BioASQ 2025: The Thirteenth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Nomic Embed: Training a Reproducible Long Context Text Embedder
• Efficient Guided Generation for Large Language Models
• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Learning To Retrieve Prompts for In-Context Learning
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Cited by

• Overview of BioASQ 2025: The Thirteenth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering

Related

• LLM Ensemble for RAG: Role of Context Length in Zero-Shot Question Answering for BioASQ Challenge
• Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach
• Using Pretrained Large Language Model with Prompt Engineering to Answer Biomedical Questions
• Can large language models reason about medical questions?
• Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks