Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future

2019/03/05 by Nan Rosemary Ke, Ke, Nan Rosemary, Amanpreet Singh +10 · 5 citations
Computer Science · #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1903.01599

openalex publication_date 2019/03/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In model-based reinforcement learning, the agent interleaves between model learning and planning. These two components are inextricably intertwined. If the model is not able to provide sensible long-term prediction, the executed planner would exploit model flaws, which can yield catastrophic failures. This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration. To this end, we build a latent-variable autoregressive model by leveraging recent ideas in variational inference. We argue that forcing latent variables to carry future information through an auxiliary task substantially improves long-term predictions. Moreover, by planning in the latent space, the planner's solution is ensured to be within regions where the model is valid. An exploration strategy can be devised by searching for unlikely trajectories under the model. Our method achieves higher reward faster compared to baselines on a variety of tasks and environments in both the imitation learning and model-based reinforcement learning settings.

Citations

• A Recurrent Latent Variable Model for Sequential Data
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Constrained model predictive control: Stability and optimality
• Prediction and Control with Temporal Segment Models
• Recurrent Environment Simulators

Cited by

• Closing the Train-Test Gap in World Models for Gradient-Based Planning
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• Active World Model Learning with Progress Curiosity
• HAEPO: History-Aggregated Exploratory Policy Optimization
• Variational Recurrent Models for Solving Partially Observable Control Tasks

Related

• Model-based Reinforcement Learning: A Survey
• Learning Dynamics Models for Model Predictive Agents
• Model-Based Reinforcement Learning via Latent-Space Collocation
• Skill-based Model-based Reinforcement Learning
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a\n Survey