Llemma: An Open Language Model For Mathematics

2023/10/16 by Zhangir Azerbayev, Azerbayev, Zhangir, Hailey Schoelkopf +15 · 2 voices · 89 citations
Computer Science · Decision Sciences · Mathematics · #Archaeology #Base (topology) #Basis (linear algebra) #Benchmark (surveying) #Code (set theory) #Computer science #Geography #Geometry #Mathematics #Mathematics, Computing, and Information Processing #Programming language #Replicate #Scientific Computing and Data Management #Statistics #Suite #Theoretical computer science #Topic Modeling #cs.AI #cs.CL #cs.LO

paper · pdf · doi:10.48550/arxiv.2310.10631

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Llemma, a large language model for mathematics. We continue pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding Llemma. On the MATH benchmark Llemma outperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is capable of tool use and formal theorem proving without any further finetuning. We openly release all artifacts, including 7 billion and 34 billion parameter models, the Proof-Pile-2, and code to replicate our experiments.

Cited by

• Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
• Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing
• AI co-mathematician: Accelerating mathematicians with agentic AI
• s1: Simple test-time scaling
• AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent
• Gödel's Poetry
• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
• Translating Informal Proofs into Formal Proofs Using a Chain of States
• MiniF2F-Dafny: LLM-Guided Mathematical Theorem Proving via Auto-Active Verification
• PCMind-2.1-Kaiyuan-2B Technical Report
• Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark
• CoRT: Code-integrated Reasoning within Thinking
• Towards Autoformalization of LLM-generated Outputs for Requirement Verification
• Stroke Modeling Enables Vectorized Character Generation with Large Vectorized Glyph Model
• RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Services
• Ghost in the Transformer: Detecting Model Reuse with Invariant Spectral Signatures
• MossNet: Mixture of State-Space Experts is a Multi-Head Attention
• MIN-Merging: Merge the Important Neurons for Model Merging
• Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspective
• Chain of Execution Supervision Promotes General Reasoning in Large Language Models
• Teaching Language Models to Reason with Tools
• Dual-Weighted Reinforcement Learning for Generative Preference Modeling
• LLM Reasoning for Machine Translation: Synthetic Data Generation over Thinking Tokens
• AutoMaAS: Self-Evolving Multi-Agent Architecture Search for Large Language Models
• Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving
• SeedPrints: Fingerprints Can Even Tell Which Seed Your Large Language Model Was Trained From
• Vocabulary Customization for Efficient Domain-Specific LLM Deployment
• Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Data
• AXIOM: A Trust-First Neuro-Symbolic Execution Architecture for Verifiable Mathematical Reasoning
• Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
• The Domain Mixed Unit: A New Neural Arithmetic Layer
• MathBuddy: A Multimodal System for Affective Math Tutoring
• Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness
• Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
• Dataset Construction for Training LLM to Learn Analog Circuit Knowledge
• Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
• MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy
• Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences
• Semantic-aware Graph-guided Behavior Sequences Generation with Large Language Models for Smart Homes
• CAMA: Enhancing Mathematical Reasoning in Large Language Models with Causal Knowledge
• LLM-Crowdsourced: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models
• A Minimalist Proof Language for Neural Theorem Proving over Isabelle/HOL
• LeanTree: Accelerating White-Box Proof Search with Factorized States in Lean 4
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms
• INTEGRALBENCH: Benchmarking LLMs with Definite Integral Problems
• Probing for Arithmetic Errors in Language Models
• Learning to Reason Across Parallel Samples for LLM Reasoning
• A Survey on Large Language Models for Mathematical Reasoning
• RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Services
• A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities
• Can Large Language Models Automate the Refinement of Cellular Network Specifications?
• Clarifying Before Reasoning: A Coq Prover with Structural Context
• LeanTutor: Towards a Verified AI Mathematical Proof Tutor
• ReasonBridge: Efficient Reasoning Transfer from Closed to Open-Source Language Models
• OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling
• Code Execution as Grounded Supervision for LLM Reasoning
• MATP-BENCH: Can MLLM Be a Good Automated Theorem Prover for Multimodal Problems?
• Can Theoretical Physics Research Benefit from Language Agents?
• Beyond Theorem Proving: Formulation, Framework and Benchmark for Formal Problem-Solving
• TokAlign: Efficient Vocabulary Adaptation via Token Alignment
• Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models
• Pruning General Large Language Models into Customized Expert Models
• Towards Generating Controllable and Solvable Geometry Problem by Leveraging Symbolic Deduction Engine
• LocalGPT: Benchmarking and Advancing Large Language Models for Local Life Services in Meituan
• Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling
• Let's Reason Formally: Natural-Formal Hybrid Reasoning Enhances LLM's Math Capability
• Why Do Some Inputs Break Low-Bit LLM Quantization?
• Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models
• Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Models
• LLM-based Automated Theorem Proving Hinges on Scalable Synthetic Data Generation
• MergeBench: A Benchmark for Merging Domain-Specialized LLMs
• Large Language Models for Computer-Aided Design: A Survey
• APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning
• Nothing from Something: Can a Language Model Discover 0?
• Learning to Repair Lean Proofs from Compiler Feedback
• A Dialogue-Based Framework for Correcting Multimodal Errors in AI-Assisted STEM Education
• RV-Syn: Rational and Verifiable Mathematical Reasoning Data Synthesis based on Structured Function Library
• Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report
• AI for Mathematics: Progress, Challenges, and Prospects
• Agentic Proof Automation: A Case Study
• NAACL2025 Tutorial: Adaptation of Large Language Models
• Foundation Models for Software Engineering of Cyber-Physical Systems: the Road Ahead
• Neural Theorem Proving: Generating and Structuring Proofs for Formal Verification
• Lightweight Latent Verifiers for Efficient Meta-Generation Strategies
• Fine-tuning a Large Language Model for Automating Computational Fluid Dynamics Simulations
• A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
• Leanabell-Prover: Posttraining Scaling in Formal Reasoning
• Foundation model [wikipedia]

Discussions

• Llemma: An Open Language Model for Mathematics [hn, 267 points, 46 comments]
• [2310.10631] Llemma: An Open Language Model For Mathematics arxiv.org/abs/2310.10631 [bsky, 0 points, 0 comments]

Related

• MathArena: Evaluating LLMs on Uncontaminated Math Competitions
• LLaMA: Open and Efficient Foundation Language Models
• Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
• Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models
• LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics