World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

2026/07/30 by Xiangcheng Zhang, Yilun Du
Computer Science · #cs.AI #cs.RO

paper · pdf

Project page at worldactionplanner.github.io

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io

Citations

• Human-level 3D shape perception emerges from multi-view learning
• Large Video Planner Enables Generalizable Robot Control
• Evaluating Gemini Robotics Policies in a Veo World Simulator
• SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models
• World Simulation with Video Foundation Models for Physical AI
• Ctrl-World: A Controllable Generative World Model for Robot Manipulation
• X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
• Video models are zero-shot learners and reasoners
• Precise Action-to-Video Generation Through Visual Action Prompts
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search
• WorldGym: World Model as An Environment for Policy Evaluation
• Inference-time Scaling of Diffusion Models through Classical Search
• DreamGen: Unlocking Generalization in Robot Learning through Video World Models
• π_0.5: a Vision-Language-Action Model with Open-World Generalization
• Wan: Open and Advanced Large-Scale Video Generative Models
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
• History-Guided Video Diffusion
• MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training
• Cosmos World Foundation Model Platform for Physical AI
• OminiControl: Minimal and Universal Control for Diffusion Transformer
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
• DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning
• RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
• IRASim: A Fine-Grained World Model for Robot Manipulation
• OpenVLA: An Open-Source Vision-Language-Action Model
• TextGrad: Automatic "Differentiation" via Text
• RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
• iVideoGPT: Interactive VideoGPTs are Scalable World Models
• Diffusion for World Modeling: Visual Details Matter in Atari
• MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
• Genie: Generative Interactive Environments
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems
• PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
• Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
• MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Video Language Planning
• Learning Interactive Real-World Simulators
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
• DINOv2: Learning Robust Visual Features without Supervision
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Improved Regret Bounds for Linear Adversarial MDPs via Linear Optimization
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• Mastering Diverse Domains through World Models
• Scalable Diffusion Models with Transformers
• The Benefits of Model-Based Generalization in Reinforcement Learning
• On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness
• Flow Matching for Generative Modeling
• Reward-Mixing MDPs with a Few Latent Contexts are Learnable
• Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximation
• Temporal Difference Learning for Model Predictive Control
• Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• On the Power of Multitask Representation Learning in Linear MDP
• Learning Transferable Visual Models From Natural Language Supervision
• Near-optimal Representation Learning for Linear Bandits and Linear RL
• When Is Generalizable Reinforcement Learning Tractable?
• Mastering Atari with Discrete World Models
• Multi-Stage Learning for Grasp-Constrained Object Manipulation with a Simulated Panda Robot
• Toward the Fundamental Limits of Imitation Learning
• Reward-Free Exploration for Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Contextual Markov Decision Processes
• Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling
• OpenAI o1 System Card

Related

• Planning with Reasoning using Vision Language World Model
• Semantic World Models
• Guiding Long-Horizon Task and Motion Planning with Vision Language Models
• GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert
• Grounding Classical Task Planners via Vision-Language Models