PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving

2025/07/10 by Mihir Parmar, Palash Goyal, Parmar, Mihir +13 · 3 citations
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2507.07495

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, decomposing complex problems into simple subtasks--a crucial part of human-like natural planning--to solve the given problem has significantly boosted the performance of large language models (LLMs). However, leveraging such planning structures during post-training to boost the performance of smaller open-source LLMs remains underexplored. Motivated by this, we introduce PLAN-TUNING, a unified post-training framework that (i) distills synthetic task decompositions (termed "planning trajectories") from large-scale LLMs and (ii) fine-tunes smaller models via supervised and reinforcement-learning objectives designed to mimic these planning processes to improve complex reasoning. On GSM8k and the MATH benchmarks, plan-tuned models outperform strong baselines by an average ∼7%. Furthermore, plan-tuned models show better generalization capabilities on out-of-domain datasets, with average ∼10% and ∼12% performance improvements on OlympiadBench and AIME 2024, respectively. Our detailed analysis demonstrates how planning trajectories improves complex reasoning capabilities, showing that PLAN-TUNING is an effective strategy for improving task-specific performance of smaller LLMs.

Citations

• AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B
• NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agents
• Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
• Is a Question Decomposition Unit All We Need?
• Large Language Models are Zero-Shot Reasoners
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Less is More: Summary of Long Instructions is Better for Program Synthesis
• Training language models to follow instructions with human feedback
• Measuring Mathematical Problem Solving With the MATH Dataset
• Proximal Policy Optimization Algorithms
• Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models

Cited by

• COMPASS: Enhancing Agent Long-Horizon Reasoning with Evolving Context
• R-Capsule: Compressing High-Level Plans for Efficient Large Language Model Reasoning
• CoreThink: A Symbolic Reasoning Layer to reason over Long Horizon Tasks with LLMs
• Cost-Awareness in Tree-Search LLM Planning: A Systematic Study

Related

• Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
• Learning to Plan with Natural Language
• Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
• Exploring and Benchmarking the Planning Capabilities of Large Language Models
• Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generation