Large Language Models for Mathematical Reasoning: Progresses and Challenges

2024/01/31 by Janice Ahn, Rishu Verma, Ahn, Janice +9 · 68 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2402.00157

openalex publication_date 2024/01/31 · openalex created_date 2024/02/04 · openalex updated_date 2026/07/28

Abstract

Mathematical reasoning serves as a cornerstone for assessing the fundamental cognitive capabilities of human intelligence. In recent times, there has been a notable surge in the development of Large Language Models (LLMs) geared towards the automated resolution of mathematical problems. However, the landscape of mathematical problem types is vast and varied, with LLM-oriented techniques undergoing evaluation across diverse datasets and settings. This diversity makes it challenging to discern the true advancements and obstacles within this burgeoning field. This survey endeavors to address four pivotal dimensions: i) a comprehensive exploration of the various mathematical problems and their corresponding datasets that have been investigated; ii) an examination of the spectrum of LLM-oriented techniques that have been proposed for mathematical problem-solving; iii) an overview of factors and concerns affecting LLMs in solving math; and iv) an elucidation of the persisting challenges within this domain. To the best of our knowledge, this survey stands as one of the first extensive examinations of the landscape of LLMs in the realm of mathematics, providing a holistic perspective on the current state, accomplishments, and future challenges in this rapidly evolving field.

Cited by

• Semantic Deception: When Reasoning Models Can't Compute an Addition
• Openpi Comet: Competition Solution For 2025 BEHAVIOR Challenge
• SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
• SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code
• Multilingual Medical Reasoning for Question Answering with Large Language Models
• Algorithmic Thinking Theory
• On GRPO Collapse in Search-R1: The Lazy Likelihood-Displacement Death Spiral
• WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate
• Rectifying LLM Thought from Lens of Optimization
• AutoMAS: A Generic Multi-Agent System for Algorithm Self-Adaptation in Wireless Networks
• TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs
• Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
• Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
• Think before Recommendation: Autonomous Reasoning-enhanced Recommender
• StreetMath: Study of LLMs' Approximation Behaviors
• The Structural Scalpel: Automated Contiguous Layer Pruning for Large Language Models
• Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning
• AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generation
• EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle
• Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Models
• Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation
• TopoAlign: A Framework for Aligning Code to Math via Topological Decomposition
• DRIFT: Decompose, Retrieve, Illustrate, then Formalize Theorems
• RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generation
• A Survey of Inductive Reasoning for Large Language Models
• ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
• RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation
• Efficient numeracy in language models through single-token number embeddings
• Iterative LLM-Based Generation and Refinement of Distracting Conditions in Math Word Problems
• BuilderBench -- A benchmark for generalist agents
• Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI
• Learning Linear Regression with Low-Rank Tasks in-Context
• More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
• A Lightweight Large Language Model-Based Multi-Agent System for 2D Frame Structural Analysis
• Read the Scene, Not the Script: Outcome-Aware Safety for LLMs
• From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMs
• Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning
• Diversity-Incentivized Exploration for Versatile Reasoning
• Personalized Auto-Grading and Feedback System for Constructive Geometry Tasks Using Large Language Models on an Online Math Platform
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
• From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervision
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
• Fast Thinking for Large Language Models
• Binary Autoencoder for Mechanistic Interpretability of Large Language Models
• SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials
• OptGraph: Large Language Models Enhanced Evolutionary Optimization Via Graph Retrieval-Augmented Generation
• Large Language Models as End-to-end Combinatorial Optimization Solvers
• CARGO: A Framework for Confidence-Aware Routing of Large Language Models
• VerilogMonkey: Exploring Parallel Scaling for Automated Verilog Code Generation with LLMs
• Gen AI in Proof-based Math Courses: A Pilot Study
• Reasoning with Preference Constraints: A Benchmark for Language Models in Many-to-One Matching Markets
• FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
• Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planning
• Astra: A Multi-Agent System for GPU Kernel Performance Optimization
• A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMs
• Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoT
• Psychometrically derived 60-question benchmarks: Substantial efficiencies and the possibility of human-AI comparisons
• Implicit Reasoning in Large Language Models: A Comprehensive Survey
• Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimization
• LegalΔ: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gain
• Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Posterior-GRPO: Rewarding Reasoning Processes in Code Generation
• Can Large Language Models Adequately Perform Symbolic Reasoning Over Time Series?
• NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty

Related

• A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges
• A Survey on Large Language Models for Mathematical Reasoning
• Large Language Models and Mathematical Reasoning Failures
• Benchmarking Large Language Models for Math Reasoning Tasks
• LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning