STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

2025/05/27 by Hossein Goli, Goli, Hossein, Michael Gimelfarb +9 · 1 citation
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2505.20781

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents over-regularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods.

Citations

• Adaptive reduced tempering For Bayesian inverse problems and rare event simulation
• Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning
• Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
• CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Making
• Policy-Guided Diffusion
• DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching
• Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
• World Models via Policy-Guided Trajectory Diffusion
• When is Offline Policy Selection Sample Efficient for Reinforcement Learning?
• Diffusion Models for Reinforcement Learning: A Survey
• Synthetic Experience Replay
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• A Review of Off-Policy Evaluation in Reinforcement Learning
• Is Conditional Generative Modeling all you need for Decision-Making?
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Planning with Diffusion for Flexible Behavior Synthesis
• Offline RL Without Off-Policy Evaluation
• Offline Reinforcement Learning as One Big Sequence Modeling Problem
• Diffusion Models Beat GANs on Image Synthesis
• Autoregressive Dynamics Models for Offline Policy Evaluation and\n Optimization
• Benchmarks for Deep Off-Policy Evaluation
• Offline Policy Selection under Uncertainty
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• Off-Policy Evaluation via the Regularized Lagrangian
• Denoising Diffusion Probabilistic Models
• MOReL : Model-Based Offline Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
• GenDICE: Generalized Offline Estimation of Stationary Values
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Learning When-to-Treat Policies
• Batch Policy Learning under Constraints
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• More Robust Doubly Robust Off-policy Evaluation
• Concentration of tempered posteriors and of their variational approximations
• SGDR: Stochastic Gradient Descent with Warm Restarts
• OpenAI Gym
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Doubly Robust Policy Evaluation and Optimization

Cited by

• Generative View Stitching
• What Do You Need for Compositional Generalization in Diffusion Planning?

Related

• State-Covering Trajectory Stitching for Diffusion Planners
• ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts
• WARPD: World model Assisted Reactive Policy Diffusion
• World Models via Policy-Guided Trajectory Diffusion
• Generative Trajectory Stitching through Diffusion Composition