DeepMDP: Learning Continuous Latent Space Models for Representation Learning

2019/06/06 by Gelada, Carles, Kumar, Saurabh, Buckman, Jacob +2 · 15 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1906.02736

Abstract

Many reinforcement learning (RL) tasks provide the agent with high-dimensional observations that can be simplified into low-dimensional continuous states. To formalize this process, we introduce the concept of a DeepMDP, a parameterized latent space model that is trained via the minimization of two tractable losses: prediction of rewards and prediction of the distribution over next latent states. We show that the optimization of these objectives guarantees (1) the quality of the latent space as a representation of the state space and (2) the quality of the DeepMDP as a model of the environment. We connect these results to prior work in the bisimulation literature, and explore the use of a variety of metrics. Our theoretical findings are substantiated by the experimental result that a trained DeepMDP recovers the latent structure underlying high-dimensional observations on a synthetic environment. Finally, we show that learning a DeepMDP as an auxiliary task in the Atari 2600 domain leads to large performance improvements over model-free RL.

Cited by

• Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning
• Spectral Representation-based Reinforcement Learning
• Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Next-Latent Prediction Transformers Learn Compact World Models
• From Embedding to Control: Representations for Stochastic Multi-Object Systems
• A Foundational Theory of Quantitative Abstraction: Adjunctions, Duality, and Logic for Probabilistic Systems
• Deep SPI: Safe Policy Improvement via World Models
• AURA: Action-Gated Memory for Robot Policies at Constant VRAM
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
• What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations
• Why and How Auxiliary Tasks Improve JEPA Representations
• Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces
• Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation
• Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalization

Related

• Low-Dimensional State and Action Representation Learning with MDP Homomorphism Metrics
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a\n Survey
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Deep Reinforcement Learning Discovers Internal Models
• DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learning