s3: You Don't Need That Much Data to Train a Search Agent via RL

2025/05/20 by Pengcheng Jiang, Jiang, Pengcheng, Xueqiang Xu +10 · 12 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.14146

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Retrieval-augmented generation (RAG) systems empower large language models (LLMs) to access external knowledge during inference. Recent advances have enabled LLMs to act as search agents via reinforcement learning (RL), improving information acquisition through multi-turn interactions with retrieval engines. However, existing approaches either optimize retrieval using search-only metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM to jointly reason and retrieve-entangling retrieval with generation and limiting the real search utility and compatibility with frozen or proprietary models. In this work, we propose s3, a lightweight, model-agnostic framework that decouples the searcher from the generator and trains the searcher using a Gain Beyond RAG reward: the improvement in generation accuracy over naive RAG. s3 requires only 2.4k training samples to outperform baselines trained on over 70x more data, consistently delivering stronger downstream performance across six general QA and five medical QA benchmarks.

Cited by

• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction
• Uncertainty Quantification for Retrieval-Augmented Reasoning
• QAgent: A modular Search Agent with Interactive Query Understanding
• Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards
• Reinforcement Learning Foundations for Deep Research Systems: A Survey
• Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering
• AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning
• MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Framework
• A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges

Related

• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning
• Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
• R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning
• ZeroSearch: Incentivize the Search Capability of LLMs without Searching