Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making

2025/12/09 by Balim, Haldun, Li, Na, Du, Yilun · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Gaussian Processes and Bayesian Inference #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2512.08280

openalex publication_date 2025/12/09 · openalex created_date 2025/12/11 · openalex updated_date 2026/07/28

Abstract

Offline decision-making via diffusion models often produces trajectories that are misaligned with system dynamics, limiting their reliability for control. We propose Model Predictive Diffuser (MPDiffuser), a compositional diffusion framework that combines a diffusion planner with a dynamics diffusion model to generate task-aligned and dynamically plausible trajectories. MPDiffuser interleaves planner and dynamics updates during sampling, progressively correcting feasibility while preserving task intent. A lightweight ranking module then selects trajectories that best satisfy task objectives. The compositional design improves sample efficiency and adaptability by enabling the dynamics model to leverage diverse and previously unseen data independently of the planner. Empirically, we demonstrate consistent improvements over prior diffusion-based methods on unconstrained (D4RL) and constrained (DSRL) benchmarks, and validate practicality through deployment on a real quadrupedal robot.

Citations

• Constrained Diffusers for Safe Planning and Control
• Latent Diffusion Planning for Imitation Learning
• Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control
• Generative Trajectory Stitching through Diffusion Composition
• Diffusion Predictive Control with Constraints
• Toward Near-Globally Optimal Nonlinear Model Predictive Control via Diffusion Models
• Diffusion Model Predictive Control
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
• Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models
• Datasets and Benchmarks for Offline Safe Reinforcement Learning
• SafeDiffuser: Safe Planning with Diffusion Probabilistic Models
• Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning
• Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
• Goal-Conditioned Imitation Learning using Score-based Diffusion Policies
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Constrained Decision Transformer for Offline Safe Reinforcement Learning
• Is Conditional Generative Modeling all you need for Decision-Making?
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Classifier-Free Diffusion Guidance
• Planning with Diffusion for Flexible Behavior Synthesis
• COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation
• Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• COMBO: Conservative Offline Model-Based Policy Optimization
• Denoising Diffusion Implicit Models
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Denoising Diffusion Probabilistic Models
• Conservative Q-Learning for Offline Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Lagrangian Neural Networks
• Reinforcement Learning in Healthcare: A Survey
• Hamiltonian Neural Networks
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Off-Policy Deep Reinforcement Learning without Exploration
• Neural Ordinary Differential Equations
• Reward Constrained Policy Optimization
• Fast Policy Learning through Imitation and Reinforcement
• World Models
• Group Normalization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• FiLM: Visual Reasoning with a General Conditioning Layer
• Proximal Policy Optimization Algorithms
• A Wavenet for Speech Denoising
• Constrained Policy Optimization
• Concrete Problems in AI Safety
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Adam: A Method for Stochastic Optimization
• On the Construction and Comparison of Difference Schemes
• On the product of semi-groups of operators

Cited by

• Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapes

Related

• Diffusion Model Predictive Control
• Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning
• Planning with Diffusion for Flexible Behavior Synthesis
• Denoising Diffusion Planner: Learning Complex Paths from Low-Quality Demonstrations
• Exploration and Adaptation in Non-Stationary Tasks with Diffusion Policies