Combating the Compounding-Error Problem with a Multi-step Model

2019/05/30 by Kavosh Asadi, Dipendra Misra, Asadi, Kavosh +4 · 7 citations
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Risk and Safety Analysis #Software Reliability and Analysis Research

paper · pdf · doi:10.48550/arxiv.1905.13320

openalex publication_date 2019/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Artificial intelligence: a modern approach
• Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning

Cited by

• Scalable Offline Model-Based RL with Action Chunks
• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• Controllable Flow Matching for Online Reinforcement Learning
• Adversarial Diffusion for Robust Reinforcement Learning
• Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• Generative Temporal Difference Learning for Infinite-Horizon Prediction
• Bounding Distributional Shifts in World Modeling through Novelty Detection

Related

• Towards a Simple Approach to Multi-step Model-based Reinforcement Learning
• Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error
• Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
• Model-based Reinforcement Learning with Multi-step Plan Value Estimation
• A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learning