Learning Markov State Abstractions for Deep Reinforcement Learning

2021/06/08 by Cameron Allen, Neev Parikh, Allen, Cameron +5 · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2106.04379

openalex publication_date 2021/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

A fundamental assumption of reinforcement learning in Markov decision processes (MDPs) is that the relevant decision process is, in fact, Markov. However, when MDPs have rich observations, agents typically learn by way of an abstract state representation, and such representations are not guaranteed to preserve the Markov property. We introduce a novel set of conditions and prove that they are sufficient for learning a Markov abstract state representation. We then describe a practical training procedure that combines inverse model estimation and temporal contrastive learning to learn an abstraction that approximately satisfies these conditions. Our novel training objective is compatible with both online and offline training: it does not require a reward signal, but agents can capitalize on reward information when available. We empirically evaluate our approach on a visual gridworld domain and a set of continuous control benchmarks. Our approach learns representations that capture the underlying structure of the domain and lead to improved sample efficiency over state-of-the-art deep reinforcement learning with visual features -- often matching or exceeding the performance achieved with hand-designed compact state information.

Citations

• Exploration by Random Network Distillation
• Provably efficient RL with Rich Observations via Latent State Decoding
• Decoupling Representation Learning from Reinforcement Learning
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Cited by

• Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning
• Multi-Agent Conditional Diffusion Model with Mean Field Communication as Wireless Resource Allocation Planner
• Pretraining in Actor-Critic Reinforcement Learning for Robot Locomotion
• From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
• One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• A Survey of State Representation Learning for Deep Reinforcement Learning

Related

• An Analysis of Model-Based Reinforcement Learning From Abstracted Observations
• Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processes
• MICo: Improved representations via sampling-based state similarity for Markov decision processes
• Learning Symbolic Representations for Reinforcement Learning of Non-Markovian Behavior
• Learning Abstract Models for Strategic Exploration and Fast Reward Transfer