FeynTune: Large Language Models for High-Energy Theory

2025/07/24 by Richmond, Paul, Agarwal, Prarit, Chowdhury, Borun +2
#Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Physical sciences #High Energy Physics - Theory (hep-th) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.03716

Abstract

We present specialized Large Language Models for theoretical High-Energy Physics, obtained as 20 fine-tuned variants of the 8-billion parameter Llama-3.1 model. Each variant was trained on arXiv abstracts (through August 2024) from different combinations of hep-th, hep-ph and gr-qc. For a comparative study, we also trained models on datasets that contained abstracts from disparate fields such as the q-bio and cs categories. All models were fine-tuned using two distinct Low-Rank Adaptation fine-tuning approaches and varying dataset sizes, and outperformed the base model on hep-th abstract completion tasks. We compare performance against leading commercial LLMs (ChatGPT, Claude, Gemini, DeepSeek) and derive insights for further developing specialized language models for High-Energy Theoretical Physics.

Citations

• Early science acceleration experiments with GPT-5
• Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics
• PhysBERT: A Text Embedding Model for Physics Scientific Literature
• The Llama 3 Herd of Models
• AstroMLab 1: Who Wins Astronomy Jeopardy!?
• A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery
• SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity
• AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets
• AstroLLaMA: Towards Specialized Foundation Models in Astronomy
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• QLoRA: Efficient Finetuning of Quantized LLMs
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• The case for 4-bit precision: k-bit Inference Scaling Laws
• Solving Quantitative Reasoning Problems with Language Models
• Building astroBERT, a language model for Astronomy & Astrophysics
• On the Opportunities and Risks of Foundation Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
• On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other

Related

• Xiwu: A Basis Flexible and Learnable LLM for High Energy Physics
• QCD in Language Models: What do they really know about QCD?
• ChatQCD: Let Large Language Models Explore QCD
• Foundation models for high-energy physics
• Towards a Large Physics Benchmark