Discriminator Augmented Model-Based Reinforcement Learning

2021/03/24 by Behzad Haghgoo, Haghgoo, Behzad, Allan Zhou +5
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Auction Theory and Applications #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2103.12999

openalex publication_date 2021/03/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

By planning through a learned dynamics model, model-based reinforcement learning (MBRL) offers the prospect of good performance with little environment interaction. However, it is common in practice for the learned model to be inaccurate, impairing planning and leading to poor performance. This paper aims to improve planning with an importance sampling framework that accounts and corrects for discrepancy between the true and learned dynamics. This framework also motivates an alternative objective for fitting the dynamics model: to minimize the variance of value estimation during planning. We derive and implement this objective, which encourages better prediction on trajectories with larger returns. We observe empirically that our approach improves the performance of current MBRL algorithms on two stochastic control problems, and provide a theoretical basis for our method.

Citations

• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Goal-Aware Prediction: Learning to Model What Matters

Related

• Model-based Reinforcement Learning: A Survey
• Planning with Expectation Models for Control
• On the role of planning in model-based deep reinforcement learning
• Learning Dynamics Models for Model Predictive Agents
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning