Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents

2025/09/03 by Paglieri, Davide, Cupiał, Bartłomiej, Cook, Jonathan +6
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.03581

Abstract

Training large language models (LLMs) to reason via reinforcement learning (RL) significantly improves their problem-solving capabilities. In agentic settings, existing methods like ReAct prompt LLMs to explicitly plan before every action; however, we demonstrate that always planning is computationally expensive and degrades performance on long-horizon tasks, while never planning further limits performance. To address this, we introduce a conceptual framework formalizing dynamic planning for LLM agents, enabling them to flexibly decide when to allocate test-time compute for planning. We propose a simple two-stage training pipeline: (1) supervised fine-tuning on diverse synthetic data to prime models for dynamic planning, and (2) RL to refine this capability in long-horizon environments. Experiments on the Crafter environment show that dynamic planning agents trained with this approach are more sample-efficient and consistently achieve more complex objectives. Additionally, we demonstrate that these agents can be effectively steered by human-written plans, surpassing their independent capabilities. To our knowledge, this work is the first to explore training LLM agents for dynamic test-time compute allocation in sequential decision-making tasks, paving the way for more efficient, adaptive, and controllable agentic systems.

Citations

• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning
• Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data
• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
• SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
• Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• s1: Simple test-time scaling
• Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
• Training Language Models to Self-Correct via Reinforcement Learning
• Automated Design of Agentic Systems
• The Llama 3 Herd of Models
• Controlling Large Language Model Agents with Entropic Activation Steering
• Chain of Thoughtlessness? An Analysis of CoT in Planning
• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
• State of What Art? A Call for Multi-Prompt LLM Evaluation
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Lost in the Middle: How Language Models Use Long Contexts
• Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach
• Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Reasoning with Language Model is Planning with World Model
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Why think step by step? Reasoning emerges from the locality of experience
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• Chain of Thought Imitation with Procedure Cloning
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• STaR: Bootstrapping Reasoning With Reasoning
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• Benchmarking the Spectrum of Agent Capabilities
• LoRA: Low-Rank Adaptation of Large Language Models
• Measuring Mathematical Problem Solving With the MATH Dataset
• An Analysis of Frame-skipping in Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Mastering Atari, Go, chess and shogi by planning with a learned model
• Learning Latent Dynamics for Planning from Pixels
• TextWorld: A Learning Environment for Text-based Games
• World Models
• Ray: A Distributed Framework for Emerging AI Applications
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Proximal Policy Optimization Algorithms
• Learning to Repeat: Fine Grained Action Repetition for Deep\n Reinforcement Learning
• The Option-Critic Architecture
• Constrained model predictive control: Stability and optimality
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• OpenAI o1 System Card

Related

• PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
• Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning
• Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach
• The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
• Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning