Deep Successor Reinforcement Learning

2016/06/08 by Tejas D. Kulkarni, Kulkarni, Tejas D., Ardavan Saeedi +5 · 1 voice · 97 citations
Computer Science · Decision Sciences · Mathematics · Neuroscience · Psychology · #Advanced Bandit Algorithms Research #Artificial intelligence #Bellman equation #Bottleneck #Computer science #Mathematical optimization #Mathematics #Neural dynamics and brain function #Psychology #Reinforcement #Reinforcement Learning in Robotics #Reinforcement learning #Social psychology #Successor cardinal #cs.AI #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1606.02396

published in arXiv (Cornell University) (Cornell University) · 10 pages, 6 figures

arxiv created 2016/06/08 · openalex publication_date 2016/06/08 · arxiv updated 2016/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor map. The successor map represents the expected future state occupancy from any given state and the reward predictor maps states to scalar rewards. The value function of a state can be computed as the inner product between the successor map and the reward weights. In this paper, we present DSR, which generalizes SR within an end-to-end deep reinforcement learning framework. DSR has several appealing properties including: increased sensitivity to distal reward changes due to factorization of reward and world dynamics, and the ability to extract bottleneck states (subgoals) given successor maps trained under a random policy. We show the efficacy of our approach on two diverse environments given raw pixel observations -- simple grid-world domains (MazeBase) and the Doom game engine.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

Cited by

• Reward-Aware Proto-Representations in Reinforcement Learning
• Constrained Reinforcement Learning Using Successor Representations
• Value Prediction Network
• Evolving Indoor Navigational Strategies Using Gated Recurrent Units In NEAT
• Spectral Representation-based Reinforcement Learning
• M³RL: Mind-aware Multi-agent Management Reinforcement Learning
• A First-Occupancy Representation for Reinforcement Learning
• A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation
• Independently Controllable Features
• A neurally plausible model learns successor representations in partially observable environments
• OpenVLN: Open-world Aerial Vision-Language Navigation
• A survey on intrinsic motivation in reinforcement learning
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments
• Shaping Belief States with Generative Environment Models for RL
• Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning
• Simple Sensor Intentions for Exploration
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• Predictive learning as a network mechanism for extracting low-dimensional latent space representations
• Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning
• Universal Successor Features Approximators
• Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
• Transfer Learning in Deep Reinforcement Learning: A Survey
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Distributional Reinforcement Learning for Multi-Dimensional Reward Functions
• Temporal Difference Learning with Neural Networks - Study of the Leakage Propagation Problem
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Autonomous Industrial Management via Reinforcement Learning: Self-Learning Agents for Decision-Making -- A Review
• Universal Successor Representations for Transfer Reinforcement Learning
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Synthesized Policies for Transfer and Adaptation across Tasks and Environments
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Dynamics-aware Embeddings
• Deep Inverse Q-learning with Constraints
• A Survey of Behavior Foundation Model: Next-Generation Whole-Body Control System of Humanoid Robots
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
• Eigenoption Discovery through the Deep Successor Representation
• Transfer in Deep Reinforcement Learning Using Successor Features and\n Generalised Policy Improvement
• MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer
• Unbiased Deep Reinforcement Learning: A General Training Framework for Existing and Future Algorithms
• A Critical Investigation of Deep Reinforcement Learning for Navigation
• MAME : Model-Agnostic Meta-Exploration
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Universal Successor Features for Transfer Reinforcement Learning

Discussions

• Deep Successor Reinforcement Learning (2016) [hn, 69 points, 2 comments]

Related

• Learning Successor Features the Simple Way
• Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning
• A neurally plausible model learns successor representations in partially observable environments
• Successor Feature Sets: Generalizing Successor Representations Across Policies
• Count-Based Exploration with the Successor Representation