Synthesizing scientific literature with retrieval-augmented language models

2026/02/04 by Akari Asai, Jacqueline He, Rulin Shao +25 · 1 voice
Medicine · Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence in Healthcare and Education #Biomedical Text Mining and Ontologies #Topic Modeling

paper · pdf · doi:10.1038/s41586-025-10072-4

openalex publication_date 2026/02/04 · openalex created_date 2026/02/06 · openalex updated_date 2026/07/29

Abstract

Scientific progress depends on the ability of researchers to synthesize the growing body of literature. Can large language models (LLMs) assist scientists in this task? Here we introduce OpenScholar, a specialized retrieval-augmented language model (LM)1 that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience and biomedicine. Despite being a smaller open model, OpenScholar-8B outperforms GPT-4o by 6.1% and PaperQA2 by 5.5% in correctness on a challenging multi-paper synthesis task from the new ScholarQABench. Although GPT-4o hallucinates citations 78–90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar’s data store, retriever and self-feedback inference loop improve off-the-shelf LMs: for instance, OpenScholar-GPT-4o improves the correctness of GPT-4o by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT-4o responses over expert-written ones 51% and 70% of the time, respectively, compared with 32% for GPT-4o. We open-source all artefacts, including our code, models, data store, datasets and a public demo. A specialized, open-source, retrieval-augmented language model is introduced for answering scientific queries and synthesizing literature, the responses of which are shown to be preferred by human evaluations over expert-written answers.

Citations

• Large language models encode clinical knowledge
• The Leaderboard Illusion
• Language agents achieve superhuman synthesis of scientific knowledge
• Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
• OpenResearcher: Unleashing AI for Accelerated Scientific Research
• The Llama 3 Herd of Models
• SciCode: A Research Coding Benchmark Curated by Scientists
• Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
• AutoSurvey: Large Language Models Can Automatically Write Surveys
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• Uni-SMART: Universal Science Multimodal Analysis and Research Transformer
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions
• LitLLM: A Toolkit for Scientific Literature Review
• Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
• Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
• SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models
• Fine-grained Hallucination Detection and Editing for Language Models
• Investigating Data Contamination for Pre-training Language Models
• The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
• Investigating Data Contamination in Modern Benchmarks for Large Language Models
• The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4
• CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
• ExpertQA: Expert-Curated Questions and Attributed Answers
• C-Pack: Packed Resources For General Chinese Embeddings
• AstroLLaMA: Towards Specialized Foundation Models in Astronomy
• Large Language Models for Automated Open-domain Scientific Hypotheses Discovery
• BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
• Lost in the Middle: How Language Models Use Long Contexts
• K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization
• A Critical Evaluation of Evaluations for Long-form Question Answering
• Enabling Large Language Models to Generate Text with Citations
• Active Retrieval Augmented Generation
• Evaluating Verifiability in Generative Search Engines
• In-Context Retrieval-Augmented Language Models
• Large Language Models Encode Clinical Knowledge
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
• Task-aware Retrieval with Instructions
• Large Language Models Struggle to Learn Long-Tail Knowledge
• RealTime QA: What's the Answer Right Now?
• BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
• Unsupervised Dense Information Retrieval with Contrastive Learning
• SciFive: a text-to-text transformer model for biomedical literature
• A Dataset of Information-Seeking Questions and Answers Anchored in\n Research Papers
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• Hurdles to Progress in Long-form Question Answering
• Multi-XScience: A Large-scale Dataset for Extreme Multi-document\n Summarization of Scientific Articles
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Fact or Fiction: Verifying Scientific Claims
• Dense Passage Retrieval for Open-Domain Question Answering
• REALM: Retrieval-Augmented Language Model Pre-Training
• PubMedQA: A Dataset for Biomedical Research Question Answering
• Passage Re-ranking with BERT
• PMC-LLaMA: Towards Building Open-source Language Models for Medicine
• The Semantic Scholar Open Data Platform

Discussions

• OpenScholar is a specialized retrieval-augmented language model that answers scientific queries by identifying relevant passages from 45 million open-access papers. It promises to not making up or ‘ha [bsky, 0 points, 0 comments]