Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
2025/01/16 by Xu, Fengli, Hao, Qianyue, Zong, Zefang +17 · 60 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2501.09686
Abstract
Language has long been conceived as an essential tool for human reasoning. The breakthrough of Large Language Models (LLMs) has sparked significant research interest in leveraging these models to tackle complex reasoning tasks. Researchers have moved beyond simple autoregressive token generation by introducing the concept of "thought" -- a sequence of tokens representing intermediate steps in the reasoning process. This innovative paradigm enables LLMs' to mimic complex human reasoning processes, such as tree search and reflective thinking. Recently, an emerging trend of learning to reason has applied reinforcement learning (RL) to train LLMs to master reasoning processes. This approach enables the automatic generation of high-quality reasoning trajectories through trial-and-error search algorithms, significantly expanding LLMs' reasoning capacity by providing substantially more training data. Furthermore, recent studies demonstrate that encouraging LLMs to "think" with more tokens during test-time inference can further significantly boost reasoning accuracy. Therefore, the train-time and test-time scaling combined to show a new research frontier -- a path toward Large Reasoning Model. The introduction of OpenAI's o1 series marks a significant milestone in this research direction. In this survey, we present a comprehensive review of recent progress in LLM reasoning. We begin by introducing the foundational background of LLMs and then explore the key technical components driving the development of large reasoning models, with a focus on automated data construction, learning-to-reason techniques, and test-time scaling. We also analyze popular open-source projects at building large reasoning models, and conclude with open challenges and future research directions.
Cited by
- LIR3AG: A Lightweight Rerank Reasoning Strategy Framework for Retrieval-Augmented Generation
- Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models
- A Rule-Aware Prompt Framework for Structured Numeric Reasoning in Cyber-Physical Systems
- Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs
- Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark
- Model Whisper: Steering Vectors Unlock Large Language Models' Potential in Test-time
- Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensics
- CRAFT-E: A Neuro-Symbolic Framework for Embodied Affordance Grounding
- Reason-Plan-ReAct: A Reasoner-Planner Supervising a ReAct Executor for Complex Enterprise Tasks
- Large Language Models as Generalist Policies for Network Optimization
- STRIDE: A Systematic Framework for Selecting AI Modalities -- Agentic AI, AI Assistants, or LLM Calls
- Evaluation of Large Language Models for Numeric Anomaly Detection in Power Systems
- HeaRT: A Hierarchical Circuit Reasoning Tree-Based Agentic Framework for AMS Design Optimization
- Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
- MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning
- Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles
- SAR-LM: Symbolic Audio Reasoning with Large Language Models
- Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond
- Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval
- When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making
- ATA: A Neuro-Symbolic Approach to Implement Autonomous and Trustworthy Agents
- FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
- Teaching Language Models to Reason with Tools
- Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
- Distractor Injection Attacks on Large Reasoning Models: Characterization and Defense
- Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
- Reducing Belief Deviation in Reinforcement Learning for Active Reasoning
- QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
- From to : Multidimensional Supervision of Reasoning Process for LLM Optimization
- Refining Hybrid Genetic Search for CVRP via Reinforcement Learning-Finetuned LLM
- Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
- ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping
- Opponent Shaping in LLM Agents
- SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations
- Large Reasoning Models Learn Better Alignment from Flawed Thinking
- Text-to-Scene with Large Reasoning Models
- AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
- Deep Thinking by Markov Chain of Continuous Thoughts
- Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning
- Agentic AI Reasoning for Mobile Edge General Intelligence: Fundamentals, Approaches, and Directions
- Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts
- Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs
- VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
- Can AI Follow In Einstein's Footsteps?
- LASAR: Latent Adaptive Semantic Aligned Reasoning for Generative Recommendation
- Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraints
- Omne-R1: Learning to Reason with Memory for Multi-hop Question Answering
- Root Cause Analysis of Radiation Oncology Incidents Using Large Language Models
- Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
- Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
- From Long to Short: LLMs Excel at Trimming Own Reasoning Chains
- A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
- Balanced Actor Initialization: Stable RLHF Training of Distillation-Based Reasoning Models
- MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers
- Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
- Hallucination vs interpretation: rethinking accuracy and precision in AI-assisted data extraction for knowledge synthesis
- Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
- FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging
- Cognitive Loop via In-Situ Optimization: Self-Adaptive Reasoning for Science
- Lessons from complex systems science for AI governance
Related