Efficient RL Training for LLMs with Experience Replay

2026/04/09 by Charles Arnal, Vivien Cabannes, Taco Cohen +2 · 1 voice · 4 citations
Computer Science · #cs.LG

paper · pdf · doi:10.48550/arxiv.2604.08706

Abstract

While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.

Citations

• Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
• EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
• LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• RL-finetuning LLMs from on- and off-policy data with a single algorithm
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
• Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
• Emergent properties with repeated examples
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
• The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
• Offline Regularised Reinforcement Learning for Large Language Models Alignment
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Measuring Mathematical Problem Solving With the MATH Dataset
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Deeper Look at Experience Replay
• Proximal Policy Optimization Algorithms
• Hindsight Experience Replay
• Safe and Efficient Off-Policy Reinforcement Learning
• Prioritized Experience Replay
• Continuous control with deep reinforcement learning
• Human-level control through deep reinforcement learning
• Self-improving reactive agents based on reinforcement learning, planning and teaching

Discussions

• CDS Silver Prof. @kempelab.bsky.social working with Charles Arnal, @tacocohen.bsky.social, Vivien Cabannes, and Remi Munos studied how LLMs can train more efficiently by reusing past experience instea [bsky, 0 points, 0 comments]

Related

• Replay For Safety
• Variance Reduction via Resampling and Experience Replay
• RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning
• Revisiting Fundamentals of Experience Replay
• Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning