Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs

2025/05/24 by Mengqi Liao, Xiangyu Xi, Liao, Mengqi +12 · 7 citations
Engineering · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Scheduling and Optimization Algorithms

paper · pdf · doi:10.48550/arxiv.2505.18573

openalex publication_date 2025/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reasoning large language models (LLMs) excel in complex tasks, which has drawn significant attention to reinforcement learning (RL) for LLMs. However, existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. This inefficiency stems from the fact that training on simple questions yields limited gains, whereas more rollouts are needed for challenging questions to sample correct answers. Furthermore, while RL improves response precision, it limits the model's exploration ability, potentially resulting in a performance cap below that of the base model prior to RL. To address these issues, we propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. Additionally, we introduce an adaptive dynamic temperature adjustment strategy to maintain the entropy at a stable level, thereby encouraging sufficient exploration. This enables LLMs to improve response precision while preserving their exploratory ability to uncover potential correct pathways. The code and data is available on: https://github.com/LiaoMengqi/E3-RL4LLMs

Cited by

• Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
• G-KV: Decoding-Time KV Cache Eviction with Global Attention
• HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
• XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
• Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
• Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning

Related

• Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
• Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
• Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective
• From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
• From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization