Bootstrap Off-policy with World Model

2025/11/01 by Zhan, Guojian, Wang, Likun, Zhang, Xiangteng +3
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2511.00423

Abstract

Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces a divergence between the collected data and the policy's actual behaviors, degrading both model learning and policy improvement. To address this, we propose BOOM (Bootstrap Off-policy with WOrld Model), a framework that tightly integrates planning and off-policy learning through a bootstrap loop: the policy initializes the planner, and the planner refines actions to bootstrap the policy through behavior alignment. This loop is supported by a jointly learned world model, which enables the planner to simulate future trajectories and provides value targets to facilitate policy improvement. The core of BOOM is a likelihood-free alignment loss that bootstraps the policy using the planner's non-parametric action distribution, combined with a soft value-weighted mechanism that prioritizes high-return behaviors and mitigates variability in the planner's action quality within the replay buffer. Experiments on the high-dimensional DeepMind Control Suite and Humanoid-Bench show that BOOM achieves state-of-the-art results in both training stability and final performance. The code is accessible at https://github.com/molumitu/BOOM_MBRL.

Citations

• Bootstrapped Model Predictive Control
• TD-M(PC)²: Improving Temporal Difference MPC Through Policy Constraint
• An Explicit Discrete-Time Dynamic Vehicle Model with Assured Numerical Stability
• HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation
• Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
• Mastering Diverse Domains through World Models
• Transformers are Sample-Efficient World Models
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Temporal Difference Learning for Model Predictive Control
• Temporal Predictive Coding For Model-Based Planning In Latent Space
• A Minimalist Approach to Offline Reinforcement Learning
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• On the role of planning in model-based deep reinforcement learning
• Mastering Atari with Discrete World Models
• Learning Off-Policy with Online Planning
• Model-Based Offline Planning
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Conservative Q-Learning for Offline Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Mastering Atari, Go, chess and shogi by planning with a learned model
• Exploring Model-based Planning with Policy Networks
• When to Trust Your Model: Model-Based Policy Optimization
• Model-Based Reinforcement Learning for Atari
• Off-Policy Deep Reinforcement Learning without Exploration
• Learning Latent Dynamics for Planning from Pixels
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• DeepMind Control Suite
• Neural Network Dynamics for Model-Based Deep Reinforcement Learning with\n Model-Free Fine-Tuning
• Distributional Soft Actor-Critic with Three Refinements

Cited by

• A KL-regularization Framework for Learning to Plan with Adaptive Priors

Related

• PWM: Policy Learning with Multi-Task World Models
• Learning Off-Policy with Online Planning
• On-Policy Model Errors in Reinforcement Learning
• A New View on Planning in Online Reinforcement Learning
• On the role of planning in model-based deep reinforcement learning