Uncertainty Prioritized Experience Replay

2025/06/10 by Rodrigo Carrasco-Davis, Carrasco-Davis, Rodrigo, Sebastian Lee +5 · 1 citation
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence in Games #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2506.09270

openalex publication_date 2025/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Prioritized experience replay, which improves sample efficiency by selecting relevant transitions to update parameter estimates, is a crucial component of contemporary value-based deep reinforcement learning models. Typically, transitions are prioritized based on their temporal difference error. However, this approach is prone to favoring noisy transitions, even when the value estimation closely approximates the target mean. This phenomenon resembles the noisy TV problem postulated in the exploration literature, in which exploration-guided agents get stuck by mistaking noise for novelty. To mitigate the disruptive effects of noise in value estimation, we propose using epistemic uncertainty estimation to guide the prioritization of transitions from the replay buffer. Epistemic uncertainty quantifies the uncertainty that can be reduced by learning, hence reducing transitions sampled from the buffer generated by unpredictable random processes. We first illustrate the benefits of epistemic uncertainty prioritized replay in two tabular toy models: a simple multi-arm bandit task, and a noisy gridworld. Subsequently, we evaluate our prioritization scheme on the Atari suite, outperforming quantile regression deep Q-learning benchmarks; thus forging a path for the use of uncertainty prioritized replay in reinforcement learning agents.

Citations

• On the Importance of Exploration for Generalization in Reinforcement Learning
• Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning
• Efficient Exploration via Epistemic-Risk-Seeking Policy Optimization
• Query The Agent: Improving sample efficiency through epistemic uncertainty estimation
• Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning
• Magnetic control of tokamak plasmas through deep reinforcement learning
• Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation
• Epistemic Neural Networks
• Posterior Meta-Replay for Continual Learning
• DEUP: Direct Epistemic Uncertainty Prediction
• How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation
• Minimum Excess Risk in Bayesian Learning
• Towards Continual Reinforcement Learning: A Review and Perspectives
• The Effectiveness of Memory Replay in Large Scale Continual Learning
• Understanding and Mitigating the Limitations of Prioritized Experience Replay
• Revisiting Fundamentals of Experience Replay
• Bandit Algorithms
• Deep Reinforcement Learning and its Neuroscientific Implications
• Reinforcement Learning and its Connections with Neuroscience and\n Psychology
• Continual Reinforcement Learning with Multi-Timescale Replay
• Never Give Up: Learning Directed Exploration Strategies
• Dota 2 with Large Scale Deep Reinforcement Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Continual Learning Using Bayesian Neural Networks
• Emergent Tool Use From Multi-Agent Autocurricula
• Experience Replay Optimization
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Meta reinforcement learning as task inference
• Safe Reinforcement Learning with Model Uncertainty Estimates
• Information-Directed Exploration for Deep Reinforcement Learning
• Episodic Curiosity through Reachability
• Randomized Prior Functions for Deep Reinforcement Learning
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Distributional Reinforcement Learning with Quantile Regression
• The Effects of Memory Replay in Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Overcoming Exploration in Reinforcement Learning with Demonstrations
• A Distributional Perspective on Reinforcement Learning
• Hindsight Experience Replay
• Curiosity-driven Exploration by Self-supervised Prediction
• Count-Based Exploration with Neural Density Models
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement\n Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning
• Deep Exploration via Bootstrapped DQN
• Prioritized Experience Replay
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPs
• Policy Gradients with Variance Related Risk Criteria
• Learning is planning: near Bayes-optimal reinforcement learning via\n Monte-Carlo tree search
• Reinforcement Learning: An Introduction

Cited by

• Reliability-Adjusted Prioritized Experience Replay

Related

• MEET: A Monte Carlo Exploration-Exploitation Trade-off for Buffer Sampling
• Temporal Difference Uncertainties as a Signal for Exploration
• Query The Agent: Improving sample efficiency through epistemic uncertainty estimation
• Exploration via Epistemic Value Estimation
• Investigating the Interplay of Prioritized Replay and Generalization