Data-Efficient Reinforcement Learning with Self-Predictive\n Representations

2020/07/12 by Max Schwarzer, Ankesh Anand, Schwarzer, Max +9 · 61 citations
Computer Science · Mathematics · #Artificial intelligence #Code (set theory) #Computer science #Domain Adaptation and Few-Shot Learning #Encoder #Machine learning #Mathematical optimization #Mathematics #Maximization #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Reinforcement learning #Sample (material) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2007.05929

published in arXiv (Cornell University) (Cornell University) · The first two authors contributed equally to this work. v4 includes new ablations and reformatting for ICLR camera ready

openalex publication_date 2020/07/12 · arxiv created 2021/05/20 · arxiv updated 2021/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

While deep reinforcement learning excels at solving tasks where large amounts\nof data can be collected through virtually unlimited interaction with the\nenvironment, learning from limited interaction remains a key challenge. We\nposit that an agent can learn more efficiently if we augment reward\nmaximization with self-supervised objectives based on structure in its visual\ninput and sequential interaction with the environment. Our method,\nSelf-Predictive Representations(SPR), trains an agent to predict its own latent\nstate representations multiple steps into the future. We compute target\nrepresentations for future states using an encoder which is an exponential\nmoving average of the agent's parameters and we make predictions using a\nlearned transition model. On its own, this future prediction objective\noutperforms prior methods for sample-efficient deep RL from pixels. We further\nimprove performance by adding data augmentation to the future prediction loss,\nwhich forces the agent's representations to be consistent across multiple views\nof an observation. Our full self-supervised objective, which combines future\nprediction and data augmentation, achieves a median human-normalized score of\n0.415 on Atari in a setting limited to 100k steps of environment interaction,\nwhich represents a 55% relative improvement over the previous state-of-the-art.\nNotably, even in this limited data regime, SPR exceeds expert human scores on 7\nout of 26 games. The code associated with this work is available at\nhttps://github.com/mila-iqia/spr\n

Citations

• Dueling Network Architectures for Deep Reinforcement Learning
• DeepMind Control Suite
• Temporal Ensembling for Semi-Supervised Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners

Cited by

• Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning
• P3: Probabilistic Policy Propagation for Stable VAE-Based Robot Learning
• Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Models
• Learning Safe Autonomous Driving Policies Using Predictive Safety Representations
• Spectral Representation-based Reinforcement Learning
• PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representations
• LAOF: Robust Latent Action Learning with Optical Flow Constraints
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Next-Latent Prediction Transformers Learn Compact World Models
• Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample\n Prediction
• Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning
• Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach
• Enhancing Tactile-based Reinforcement Learning for Robotic Control
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
• Decoupling Representation Learning from Reinforcement Learning
• Dual Goal Representations
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
• DyMoDreamer: World Modeling with Dynamic Modulation
• Generalization in Reinforcement Learning by Soft Data Augmentation
• TAPO: Transition-Aware Policy Optimization for LLM Agents
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• What Fundamental Structure in Reward Functions Enables Efficient Sparse-Reward Learning?
• Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning
• Learning State Representations from Random Deep Action-conditional Predictions
• Stackelberg Coupling of Online Representation Learning and Reinforcement Learning
• Learning Representations for Pixel-based Control: What Matters and Why?
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning
• Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
• Hierarchical Planning with Latent World Models
• Spectral Bellman Method: Unifying Representation and Exploration in RL
• BYOL works even without batch statistics
• Intention-Conditioned Flow Occupancy Models
• Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease
• A Survey of State Representation Learning for Deep Reinforcement Learning
• Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Self-Predictive Dynamics for Generalization of Vision-based Reinforcement Learning
• seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models
• Simple, Good, Fast: Self-Supervised World Models Free of Baggage
• Descriptive History Representations: Learning Representations by Answering Questions
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
• Universal Value-Function Uncertainties
• Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Hadamax Encoding: Elevating Performance in Model-Free Atari
• FLARE: Robot Learning with Implicit World Modeling
• Improving Few-Shot Learning with Auxiliary Self-Supervised Pretext Tasks
• DSADF: Thinking Fast and Slow for Decision Making
• Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
• Reinforcement Learning with Latent Flow
• Stable Deep Reinforcement Learning via Isotropic Gaussian Representations
• Next Embedding Prediction Makes World Models Stronger
• Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning
• Revisiting TD Target Aggregation under Uncertainty in Q-Learning
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
• Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control

Related

• Reward prediction for representation learning and reward shaping
• Understanding Self-Predictive Learning for Reinforcement Learning
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL
• Dynamics-aware Embeddings