Recall Traces: Backtracking Models for Efficient Reinforcement Learning

2018/04/02 by Anirudh Goyal, Goyal, Anirudh, Philémon Brakel +14 · 1 voice · 5 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1804.00379

Accepted at ICLR 2019

openalex publication_date 2018/04/02 · arxiv published 2018/04/02 · arxiv created 2019/01/28 · arxiv updated 2019/01/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the use of a backtracking model that predicts the preceding states that terminate at a given high-reward state. We can train a model which, starting from a high value state (or one that is estimated to have high value), predicts and sample for which the (state, action)-tuples may have led to that high value state. These traces of (state, action) pairs, which we refer to as Recall Traces, sampled from this backtracking model starting from a high value state, are informative as they terminate in good states, and hence we can use these traces to improve a policy. We provide a variational interpretation for this idea and a practical algorithm in which the backtracking model samples from an approximate posterior distribution over trajectories which lead to large rewards. Our method improves the sample efficiency of both on- and off-policy RL algorithms across several environments and tasks.

Citations

• Learning Continuous Control Policies by Stochastic Value Gradients
• Reinforcement learning in robotics: A survey
• Continuous control with deep reinforcement learning
• End to End Learning for Self-Driving Cars
• Unifying Count-Based Exploration and Intrinsic Motivation
• Combining policy gradient and Q-learning
• Sample Efficient Actor-Critic with Experience Replay
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Count-Based Exploration with Neural Density Models
• Recurrent Environment Simulators
• Automatic Goal Generation for Reinforcement Learning Agents
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Cited by

• Offline Reinforcement Learning with Reverse Model-based Imagination
• Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning
• Imitating Latent Policies from Observation
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning

Discussions

• Recall Traces: Backtracking Models for Efficient Reinforcement Learning [hn, 2 points, 0 comments]

Related

• Learning State Representations via Retracing in Reinforcement Learning
• Partially Observable Reinforcement Learning with Memory Traces
• SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning
• Bootstrapped Representations in Reinforcement Learning
• Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks