Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning

2025/05/27 by Mingyang Song, Song, Mingyang, Mao Zheng +1 · 7 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation

paper · pdf · doi:10.48550/arxiv.2505.21178

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As test-time scaling becomes a pivotal research frontier in Large Language Models (LLMs) development, contemporary and advanced post-training methodologies increasingly focus on extending the generation length of long Chain-of-Thought (CoT) responses to enhance reasoning capabilities toward DeepSeek R1-like performance. However, recent studies reveal a persistent overthinking phenomenon in state-of-the-art reasoning models, manifesting as excessive redundancy or repetitive thinking patterns in long CoT responses. To address this issue, in this paper, we propose a simple yet effective two-stage reinforcement learning framework for achieving concise reasoning in LLMs, named ConciseR. Specifically, the first stage, using more training steps, aims to incentivize the model's reasoning capabilities via Group Relative Policy Optimization with clip-higher and dynamic sampling components (GRPO++), and the second stage, using fewer training steps, explicitly enforces conciseness and improves efficiency via Length-aware Group Relative Policy Optimization (L-GRPO). Significantly, ConciseR only optimizes response length once all rollouts of a sample are correct, following the "walk before you run" principle. Extensive experimental results demonstrate that our ConciseR model, which generates more concise CoT reasoning responses, outperforms recent state-of-the-art reasoning models with zero RL paradigm across AIME 2024, MATH-500, AMC 2023, Minerva, and Olympiad benchmarks.

Citations

• A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
• Concise Reasoning via Reinforcement Learning
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
• Understanding R1-Zero-Like Training: A Critical Perspective
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond
• The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks
• Process Reinforcement through Implicit Rewards
• s1: Simple test-time scaling
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Qwen2.5 Technical Report
• Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
• Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• Solving Quantitative Reasoning Problems with Language Models
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Measuring Mathematical Problem Solving With the MATH Dataset
• Proximal Policy Optimization Algorithms
• OpenAI o1 System Card

Cited by

• OckBench: Measuring the Efficiency of LLM Reasoning
• Concise Reasoning in the Lens of Lagrangian Optimization
• Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
• ThinkBrake: Mitigating Overthinking in Tool Reasoning
• OptimalThinkingBench: Evaluating Over and Underthinking in LLMs
• Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
• ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
• When Less is Enough: Efficient Inference via Collaborative Reasoning

Related

• Rectifying LLM Thought from Lens of Optimization
• Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
• Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accurately
• Demystifying Long Chain-of-Thought Reasoning in LLMs
• When More is Less: Understanding Chain-of-Thought Length in LLMs