Self-Consistent Models and Values

2021/10/25 by Gregory Farquhar, Farquhar, Gregory, Kate Baumli +11
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Reinforcement Learning in Robotics #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2110.12840

NeurIPS 2021

arxiv created 2021/10/25 · openalex publication_date 2021/10/25 · arxiv updated 2021/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a learned model and value function to be jointly self-consistent. Our approach differs from classic planning methods such as Dyna, which only update values to be consistent with the model. We propose multiple self-consistency updates, evaluate these in both tabular and function approximation settings, and find that, with appropriate choices, self-consistency helps both policy evaluation and control.

Citations

• A Markovian Decision Process
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Value Prediction Network
• Imagination-Augmented Agents for Deep Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
• Observe and Look Further: Achieving Consistent Performance on Atari
• Shaping Belief States with Generative Environment Models for RL
• Understanding Back-Translation at Scale
• Dream to Control: Learning Behaviors by Latent Imagination
• Causally Correct Partial Models for Reinforcement Learning
• Value-driven Hindsight Modelling
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Online and Offline Reinforcement Learning by Planning with a Learned Model

Related

• The Value Equivalence Principle for Model-Based Reinforcement Learning
• Should Models Be Accurate?
• Learning Powerful Policies by Using Consistent Dynamics Model
• Planning with Expectation Models for Control
• Simplified Temporal Consistency Reinforcement Learning