RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning

2025/07/10 by Hongzhi Zhang, Zhang, Hongzhi, Jia Fu +11 · 11 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.07451

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) for large language models is an energy-intensive endeavor: training can be unstable, and the policy may gradually drift away from its pretrained weights. We present RLEP -- Reinforcement Learning with Experience rePlay -- a two-phase framework that first collects verified trajectories and then replays them during subsequent training. At every update step, the policy is optimized on mini-batches that blend newly generated rollouts with these replayed successes. By replaying high-quality examples, RLEP steers the model away from fruitless exploration, focuses learning on promising reasoning paths, and delivers both faster convergence and stronger final performance. On the Qwen2.5-Math-7B base model, RLEP reaches baseline peak accuracy with substantially fewer updates and ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%, on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our code, datasets, and checkpoints are publicly available at https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further research.

Citations

• EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• Qwen3 Technical Report
• Learning to Reason under Off-Policy Guidance
• Improving RL Exploration for LLM Reasoning through Retrospective Replay
• Understanding R1-Zero-Like Training: A Critical Perspective
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Prioritized Experience Replay

Cited by

• Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
• ExGRPO: Learning to Reason from Experience
• XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
• Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
• SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
• DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents
• RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
• DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training
• EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding
• R³: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learning

Related

• Improving RL Exploration for LLM Reasoning through Retrospective Replay
• SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
• Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
• Reinforcement Learning on Pre-Training Data