Dynamics-aware Embeddings

2019/08/25 by William Whitney, WILLIAM F. WHITNEY, Whitney, William +6
Computer Science · Decision Sciences · Mathematics · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Simulation Techniques and Applications #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1908.09357

Published at ICLR 2020

openalex publication_date 2019/08/25 · arxiv created 2020/01/14 · arxiv updated 2020/01/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL). We propose a forward prediction objective for simultaneously learning embeddings of states and action sequences. These embeddings capture the structure of the environment's dynamics, enabling efficient policy learning. We demonstrate that our action embeddings alone improve the sample efficiency and peak performance of model-free RL on control from low-dimensional states. By combining state and action embeddings, we achieve efficient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.

Citations

• Adam: A Method for Stochastic Optimization
• The information bottleneck method
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Continuous control with deep reinforcement learning
• Deep Successor Reinforcement Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• DeepMind Control Suite
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Addressing Function Approximation Error in Actor-Critic Methods
• Distributed Distributional Deterministic Policy Gradients
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• Soft Actor-Critic Algorithms and Applications
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• The Natural Language of Actions
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

Related

• Learning Action Representations for Reinforcement Learning
• For SALE: State-Action Representation Learning for Deep Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive\n Representations
• State Representation Learning for Goal-Conditioned Reinforcement Learning
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning