Unsupervised State Representation Learning in Atari

2019/06/19 by Ankesh Anand, Evan Racah, Anand, Ankesh +9 · 9 citations
Computer Science · #Artificial Intelligence in Games #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1906.08226

openalex publication_date 2019/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

State representation learning, or the ability to capture latent generative factors of an environment, is crucial for building intelligent agents that can perform a wide variety of tasks. Learning such representations without supervision from rewards is a challenging open problem. We introduce a method that learns state representations by maximizing mutual information across spatially and temporally distinct features of a neural encoder of the observations. We also introduce a new benchmark based on Atari 2600 games where we evaluate representations based on how well they capture the ground truth state variables. We believe this new framework for evaluating representation learning models will be crucial for future representation learning research. Finally, we compare our technique with other state-of-the-art generative and contrastive representation learning methods. The code associated with this work is available at https://github.com/mila-iqia/atari-representation-learning

Citations

• Auto-Encoding Variational Bayes
• Representation Learning: A Review and New Perspectives
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning deep representations by mutual information estimation and maximization
• Recurrent World Models Facilitate Policy Evolution
• Towards a Definition of Disentangled Representations
• Learning Representations by Maximizing Mutual Information Across Views
• Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
• Revisiting Self-Supervised Visual Representation Learning

Cited by

• Spectral Representation-based Reinforcement Learning
• Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning
• Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
• Enhancing Tactile-based Reinforcement Learning for Robotic Control
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Mutual Information Tracks Policy Coherence in Reinforcement Learning
• STORI: A Benchmark and Taxonomy for Stochastic Environments
• Stackelberg Coupling of Online Representation Learning and Reinforcement Learning
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

Related

• Unsupervised Representation Learning in Partially Observable Atari Games
• Pretrained Encoders are All You Need
• Playing Atari with Six Neurons
• A Survey of State Representation Learning for Deep Reinforcement Learning
• Learning and Querying Fast Generative Models for Reinforcement Learning