LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
2024/09/20 by Karthik Valmeekam, Kaya Stechly, Valmeekam, Karthik +3 · 6 voices · 40 citations
Computer Science · #Archaeology #Computer science #History #Library Science and Information Systems #Mathematics, Computing, and Information Processing #Plan (archaeology) #Semantic Web and Ontologies #cs.AI #cs.CL
paper · pdf · doi:10.48550/arxiv.2409.13373
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities. PlanBench, an extensible benchmark we developed in 2022, soon after the release of GPT3, has remained an important tool for evaluating the planning abilities of LLMs. Despite the slew of new private and open source LLMs since GPT3, progress on this benchmark has been surprisingly slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs--making it a new kind of model: a Large Reasoning Model (LRM). Using this development as a catalyst, this paper takes a comprehensive look at how well current LLMs and new LRMs do on PlanBench. As we shall see, while o1's performance is a quantum improvement on the benchmark, outpacing the competition, it is still far from saturating it. This improvement also brings to the fore questions about accuracy, efficiency, and guarantees which must be considered before deploying such systems.
Cited by
- Fluid Reasoning Representations
- Reasoning Models Reason Well, Until They Don't
- Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought
- On the Limits of LLM Reasoning: Evidence From Contamination, Translation, and Answer Modification in Multiple-Choice Benchmarks
- Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation
- SymStep: Symbolic Step Verification for Logical Reasoning
- SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
- CoS: Towards Optimal Event Scheduling via Chain-of-Scheduling
- Automating Benchmark Design
- TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games
- Language Model as Planner and Formalizer under Constraints
- Toward PDDL Planning Copilot
- Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees
- Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling
- CoreThink: A Symbolic Reasoning Layer to reason over Long Horizon Tasks with LLMs
- Weisfeiler-Leman Features for Planning: A 1,000,000 Sample Size Hyperparameter Study
- Language Models For Generalised PDDL Planning: Synthesising Sound and Programmatic Policies
- Inspire or Predict? Exploring New Paradigms in Assisting Classical Planners with Large Language Models
- The Missing Reward: Active Inference in the Era of Experience
- Mockingbird: How does LLM perform in general machine learning tasks?
- Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
- On the Relevance of Logic for Artificial Intelligence, and the Promise of Neurosymbolic Learning
- Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories
- Advancing Multi-Step Mathematical Reasoning in Large Language Models through Multi-Layered Self-Reflection with Auto-Prompting
- Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs
- Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows
- Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents
- The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models
- Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors
- Rethinking the Unsolvable: When In-Context Search Meets Test-Time Scaling
- Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Risks
- Are LLMs Better Formalizers than Solvers on Complex Problems?
- Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective
- ChatHTN: Interleaving Approximate (LLM) and Symbolic HTN Planning
- Reasoning Large Language Model Errors Arise from Hallucinating Critical Problem Features
- Learning Virtual Machine Scheduling in Cloud Computing through Language Agents
- LLMs versus the Halting Problem: Characterizing Program Termination Reasoning
- Evaluating the Goal-Directedness of Large Language Models
Discussions
- TIL: they’re only called LLMs if they come from the non-reasoning region of france arxiv.org/abs/2409.13373 [bsky, 62 points, 2 comments]
- New paper: "LLMs still Can't Plan: Can LRMs? A Preliminary Evaluation on OpenAI's o1 on PlanBench. LRM = "Large Reasoning Model" arxiv.org/abs/2409.133... [bsky, 10 points, 5 comments]
- New paper analyzes o1’s reasoning capabilities so now AI gaslights? arxiv.org/abs/2409.13373 [bsky, 2 points, 1 comments]
- LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of O1 on PlanBench [hn, 2 points, 1 comments]
- But you do see interesting research that raises the problem in new lights, even where there are capability gains. When @rao2z.bsky.social's ASU team tested o1-preview, they noted the capability gains [bsky, 1 points, 1 comments]
- So LRMs (reasoning models) are LLMs with more entropy (random shit choices)? https://arxiv.org/abs/2409.13373 [bsky, 0 points, 0 comments]
Related