Safe and Efficient Off-Policy Reinforcement Learning

2016/06/08 by Rémi Munos, Tom Stepleton, Munos, Rémi +5 · 1 voice · 25 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Formal Methods in Verification #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1606.02647

openalex publication_date 2016/06/08 · arxiv published 2016/06/08 · arxiv updated 2016/11/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace(λ), with three desired properties: (1) it has low variance; (2) it safely uses samples collected from any behaviour policy, whatever its degree of "off-policyness"; and (3) it is efficient as it makes the best use of samples collected from near on-policy behaviour policies. We analyze the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. We believe this is the first return-based off-policy control algorithm converging a.s. to Q^* without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins' Q(λ), which was an open problem since 1989. We illustrate the benefits of Retrace(λ) on a standard suite of Atari 2600 games.

Cited by

• Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning
• Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates
• Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition
• Conformal Constrained Policy Optimization for Cost-Effective LLM Agents
• KAN/H: Kolmogorov-Arnold Network using Haar-like bases
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning
• Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning
• Algorithm-Relative Trajectory Valuation in Policy Gradient Control
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
• An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes
• PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
• Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI
• Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
• Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark
• Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalization
• Machine learning for reliability engineering and safety applications: Review of current status and future opportunities
• Spectral Bellman Method: Unifying Representation and Exploration in RL
• Sample Efficient Actor-Critic with Experience Replay
• A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
• Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs
• Experimenting, Fast and Slow: Bayesian Optimization of Long-term Outcomes with Online Experiments

Discussions

• Currently experimenting with exploration policies for deep RL on Super Mario Bros - Agent is able to beat all levels I threw at it - You can watch the AI learn live [lemmy, 3 points, 0 comments]

Related

• Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift
• Offline RL Without Off-Policy Evaluation
• Efficient Offline Reinforcement Learning: First Imitate, then Improve
• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
• Adaptive Trade-Offs in Off-Policy Learning