Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning

2025/09/26 by Xiaoyu Tan, Qin, Yulei, Tan, Xiaoyu +27 · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics #Robot Manipulation and Learning

paper · pdf · doi:10.48550/arxiv.2509.22601

openalex publication_date 2025/09/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) is the dominant paradigm for sharpening strategic tool use capabilities of LLMs on long-horizon, sparsely-rewarded agent tasks, yet it faces a fundamental challenge of exploration-exploitation trade-off. Existing studies stimulate exploration through the lens of policy entropy, but such mechanical entropy maximization is prone to RL instability due to the multi-turn distribution shifting. In this paper, we target the progressive exploration-exploitation balance under the guidance of the agent's own experiences without succumbing to either entropy collapsing or runaway divergence. We propose SPEAR, a self-imitation learning (SIL) recipe for training agentic LLMs. It extends the vanilla SIL, where a replay buffer stores good experience for off-policy update, by gradually steering the policy entropy across stages. Specifically, the proposed curriculum scheduling harmonizes intrinsic reward shaping and self-imitation to 1) expedite exploration via frequent tool interactions at the beginning, and 2) strengthen exploitation of successful tactics upon convergence towards familiarity with the environment. We also combine bag-of-tricks of industrial RL optimizations for a strong baseline Dr.BoT to demonstrate our effectiveness. In ALFWorld and WebShop, SPEAR increases the success rates of GRPO/GiGPO/Dr.BoT by up to 16.1%/5.1%/8.6% and 20.7%/11.8%/13.9%, respectively. In AIME24 and AIME25, SPEAR boosts Dr.BoT by up to 3.8% and 6.1%, respectively. Such gains incur only 10%-25% extra theoretical complexity and negligible runtime overhead in practice, demonstrating the plug-and-play scalability of SPEAR.

Citations

• Agentic Reinforcement Learning with Implicit Step Rewards
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
• Understanding Tool-Integrated Reasoning
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewards
• Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
• Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL
• RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
• AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
• Agentic Reinforced Policy Optimization
• Group Sequence Policy Optimization
• MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation
• WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization
• Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models
• Towards Effective Code-Integrated Reasoning
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
• An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
• Group-in-Group Policy Optimization for LLM Agent Training
• Qwen3 Technical Report
• Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving
• WebThinker: Empowering Large Reasoning Models with Deep Research Capability
• Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• ToolRL: Reward is All Tool Learning Needs
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
• ToRL: Scaling Tool-Integrated RL
• Understanding R1-Zero-Like Training: A Critical Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• AgentRM: Enhancing Agent Generalization with Reward Modeling
• Qwen2.5-VL Technical Report
• Process Reinforcement through Implicit Rewards
• s1: Simple test-time scaling
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• FullStack Bench: Evaluating LLMs as Full Stack Coders
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
• Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent
• How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective
• Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models
• Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
• Teaching Large Language Models to Reason with Reinforcement Learning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
• CogAgent: A Visual Language Model for GUI Agents
• Tool-Augmented Reward Modeling
• ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Multimodal Web Navigation with Instruction-Finetuned Foundation Models
• Reflexion: Language Agents with Verbal Reinforcement Learning
• GPT-4 Technical Report
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
• Measuring and Narrowing the Compositionality Gap in Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Actor Prioritized Experience Replay
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
• Self-Imitation Learning from Demonstrations
• Training language models to follow instructions with human feedback
• Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions
• MuSiQue: Multihop Questions via Single-hop Question Composition
• Self-Imitation Learning by Planning
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Self-Imitation Advantage Learning
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Understanding and Mitigating the Limitations of Prioritized Experience Replay
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Exploration versus exploitation in reinforcement learning: a stochastic control approach
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• TextWorld: A Learning Environment for Text-based Games
• Self-Imitation Learning
• Learning Self-Imitating Diverse Policies
• Diversity is All You Need: Learning Skills without a Reward Function
• Distributed Prioritized Experience Replay
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• Curiosity-driven Exploration by Self-supervised Prediction
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• Equivalence Between Policy Gradients and Soft Q-Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Variational Intrinsic Control
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement\n Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Prioritized Experience Replay
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Power
• Practical Bayesian Optimization of Machine Learning Algorithms
• Adaptive Mixtures of Local Experts
• Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design
• OpenAI o1 System Card

Cited by

• SmartSnap: Proactive Evidence Seeking for Self-Verifying Agents

Related

• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• Demystifying Reinforcement Learning in Agentic Reasoning
• Self-Imitation Learning
• Agentic Policy Optimization via Instruction-Policy Co-Evolution
• SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning