Achieving GPT-4o level performance in astronomy with a specialized 8B-parameter large language model

2024/11/13 by Tijmen de Haan, T. de Haan, Yuan-Sen Ting +8 · 2 voices
Computer Science · Engineering · Mathematics · #Artificial intelligence #Astronomy #Benchmark (surveying) #Class (philosophy) #Computer science #Cosmology #Domain (mathematical analysis) #Engineering #Geography #Instrumentation (computer programming) #Mathematics #Mathematics, Computing, and Information Processing #Natural Language Processing Techniques #Physics #Programming language #Range (aeronautics) #Topic Modeling

paper · pdf · doi:10.1038/s41598-025-97131-y

openalex publication_date 2025/04/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

AstroSage-Llama-3.1-8B is a domain-specialized natural-language AI assistant tailored for research in astronomy, astrophysics, cosmology, and astronomical instrumentation. Trained on the complete collection of astronomy-related arXiv papers from 2007 to 2024 along with millions of synthetically-generated question-answer pairs and other astronomical literature, AstroSage-Llama-3.1-8B demonstrates remarkable proficiency on a wide range of questions. AstroSage-Llama-3.1-8B scores 80.9% on the AstroMLab-1 benchmark, greatly outperforming all models-proprietary and open-weight-in the 8-billion parameter class, and performing on par with GPT-4o. This achievement demonstrates the potential of domain specialization in AI, suggesting that focused training can yield capabilities exceeding those of much larger, general-purpose models. AstroSage-Llama-3.1-8B is freely available, enabling widespread access to advanced AI capabilities for astronomical education and research.

Discussions

• arxiv preprints that caught my eye today: “The Last Arecibo Message” arxiv.org/abs/2411.09790 “Discovery of a Rare Group of Dwarf Galaxies in the Local Universe” arxiv.org/abs/2411.10045 “Fast Radio [bsky, 17 points, 2 comments]
• Sometimes bigger models are better… sometimes domain specific models are better! arxiv.org/abs/2411.09012 [bsky, 1 points, 0 comments]

Related

• AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model
• AstroMLab 1: Who Wins Astronomy Jeopardy!?
• AstroLLaMA: Towards Specialized Foundation Models in Astronomy
• Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)
• AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets