Evaluating Retrieval-Augmented Generation Agents for Autonomous Scientific Discovery in Astrophysics

2025/07/09 by Xueqing Xu, Boris Bolliet, Xu, Xueqing +10 · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computational and Text Analysis Methods #Cosmology and Nongalactic Astrophysics (astro-ph.CO) #FOS: Computer and information sciences #FOS: Physical sciences #Instrumentation and Methods for Astrophysics (astro-ph.IM) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.07155

openalex publication_date 2025/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We evaluate 9 Retrieval Augmented Generation (RAG) agent configurations on 105 Cosmology Question-Answer (QA) pairs that we built specifically for this purpose.The RAG configurations are manually evaluated by a human expert, that is, a total of 945 generated answers were assessed. We find that currently the best RAG agent configuration is with OpenAI embedding and generative model, yielding 91.4% accuracy. Using our human evaluation results we calibrate LLM-as-a-Judge (LLMaaJ) system which can be used as a robust proxy for human evaluation. These results allow us to systematically select the best RAG agent configuration for multi-agent system for autonomous scientific discovery in astrophysics (e.g., cmbagent presented in a companion paper) and provide us with an LLMaaJ system that can be scaled to thousands of cosmology QA pairs. We make our QA dataset, human evaluation results, RAG pipelines, and LLMaaJ system publicly available for further use by the astrophysics community.

Citations

• The Atacama Cosmology Telescope: DR6 Constraints on Extended Cosmological Models
• What is the Role of Large Language Models in the Evolution of Astronomy Research?
• Language agents achieve superhuman synthesis of scientific knowledge
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomy
• A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery
• Designing an Evaluation Framework for Large Language Models in Astronomy Research
• Dated Data: Tracing Knowledge Cutoffs in Large Language Models
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy
• MTEB: Massive Text Embedding Benchmark
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networks
• A large annotated corpus for learning natural language inference

Cited by

• Seeing the Forest Through the Trees: Knowledge Retrieval for Streamlining Particle Physics Analysis

Related

• AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model
• MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoning
• PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
• SQuAI: Scientific Question-Answering with Multi-Agent Retrieval-Augmented Generation
• From Queries to Criteria: Understanding How Astronomers Evaluate LLMs