Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

2025/10/02 by Dou, Zhihao, Zhao, Qinjian, Wan, Zhongwei +9 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.01833

Abstract

Large language models (LLMs) have demonstrated remarkable reasoning abilities in complex tasks, often relying on Chain-of-Thought (CoT) reasoning. However, due to their autoregressive token-level generation, the reasoning process is largely constrained to local decision-making and lacks global planning. This limitation frequently results in redundant, incoherent, or inaccurate reasoning, which significantly degrades overall performance. Existing approaches, such as tree-based algorithms and reinforcement learning (RL), attempt to address this issue but suffer from high computational costs and often fail to produce optimal reasoning trajectories. To tackle this challenge, we propose Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization PTA-GRPO, a two-stage framework designed to improve both high-level planning and fine-grained CoT reasoning. In the first stage, we leverage advanced LLMs to distill CoT into compact high-level guidance, which is then used for supervised fine-tuning (SFT). In the second stage, we introduce a guidance-aware RL method that jointly optimizes the final output and the quality of high-level guidance, thereby enhancing reasoning effectiveness. We conduct extensive experiments on multiple mathematical reasoning benchmarks, including MATH, AIME2024, AIME2025, and AMC, across diverse base models such as Qwen2.5-7B-Instruct, Qwen3-8B, Qwen3-14B, and LLaMA3.2-3B. Experimental results demonstrate that PTA-GRPO consistently achieves stable and significant improvements across different models and tasks, validating its effectiveness and generalization.

Citations

• TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
• OpenThoughts: Data Recipes for Reasoning Models
• Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
• SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
• ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay
• SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
• Group-in-Group Policy Optimization for LLM Agent Training
• Qwen3 Technical Report
• DSADF: Thinking Fast and Slow for Decision Making
• SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
• Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
• Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
• Rethinking Reflection in Pre-Training
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
• A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond
• Understanding R1-Zero-Like Training: A Critical Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning
• Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks
• MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning
• Qwen2.5-VL Technical Report
• Process Reinforcement through Implicit Rewards
• Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
• DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
• LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning
• CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
• The Llama 3 Herd of Models
• LiteSearch: Efficacious Tree Search for LLM
• A Survey on Large Language Models for Code Generation
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• Improving Large Language Model Fine-tuning for Solving Math Problems
• Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training
• MathChat: Converse to Tackle Challenging Math Problems with LLM Agents
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
• An Extensible Multimodal Multi-task Object Dataset with Materials
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Proximal Policy Optimization Algorithms
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

Cited by

• CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

Related

• Reasoning with Language Model is Planning with World Model
• CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
• rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection
• DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
• Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation