Reliability-Adjusted Prioritized Experience Replay

2025/06/23 by Pleiss, Leonard S., Sutter, Tobias, Schiffer, Maximilian
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2506.18482

Abstract

Experience replay enables data-efficient learning from past experiences in online reinforcement learning agents. Traditionally, experiences were sampled uniformly from a replay buffer, regardless of differences in experience-specific learning potential. In an effort to sample more efficiently, researchers introduced Prioritized Experience Replay (PER). In this paper, we propose an extension to PER by introducing a novel measure of temporal difference error reliability. We theoretically show that the resulting transition selection algorithm, Reliability-adjusted Prioritized Experience Replay (ReaPER), enables more efficient learning than PER. We further present empirical results showing that ReaPER outperforms PER across various environment types, including the Atari-10 benchmark.

Citations

• Uncertainty Prioritized Experience Replay
• Investigating the Interplay of Prioritized Replay and Generalization
• EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data
• Bigger, Better, Faster: Human-level Atari with human-level efficiency
• Mastering Diverse Domains through World Models
• Atari-5: Distilling the Arcade Learning Environment down to Five Games
• Mastering Atari Games with Limited Data
• Learning to Sample with Local and Global Contexts in Experience Replay Buffer
• Revisiting Fundamentals of Experience Replay
• Agent57: Outperforming the Atari Human Benchmark
• Never Give Up: Learning Directed Exploration Strategies
• Mastering Atari, Go, chess and shogi by planning with a learned model
• Experience Replay Optimization
• Prioritized Sequence Experience Replay
• Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replay
• Go-Explore: a New Approach for Hard-Exploration Problems
• Experience Replay for Continual Learning
• Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update
• Selective Experience Replay for Lifelong Learning
• Distributed Prioritized Experience Replay
• A Deeper Look at Experience Replay
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Hindsight Experience Replay
• An overview of gradient descent optimization algorithms
• Prioritized Experience Replay
• Deep Reinforcement Learning with Double Q-learning
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Playing Atari with Deep Reinforcement Learning
• Reinforcement Learning: An Introduction

Related

• Prioritized Experience Replay
• Prioritized Sequence Experience Replay
• Improving Experience Replay with Successor Representation
• Experience Replay Optimization
• Revisiting Fundamentals of Experience Replay