Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning

2025/05/16 by Yaorui Shi, Sihang Li, Shi, Yaorui +13 · 21 citations
Computer Science · #AI-based Problem Solving and Planning #Intelligent Tutoring Systems and Adaptive Learning #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2505.11277

Abstract

Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir. Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning. In this paper, we propose AutoRefine, a reinforcement learning post-training framework that adopts a new "search-and-refine-during-think" paradigm. AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer. Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization. Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.

Cited by

• Laser: Governing Long-Horizon Agentic Search via Structured Protocol and Context Register
• CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning
• HuggingR⁴: A Progressive Reasoning Framework for Discovering Optimal Model Companions
• Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning
• Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models
• MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning
• InfoFlow: Reinforcing Search Agent Via Reward Density Optimization
• Rethinking Self-Evolution: A Constrained Exploration-Exploitation Process for Mitigating Skill Overfitting
• Rethinking the Design of Reinforcement Learning-Based Deep Research Agents
• GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learning
• LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts
• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle
• Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation
• Understanding DeepResearch via Reports
• Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents
• Do LLM Agents Know How to Ground, Recover, and Assess? A Benchmark for Epistemic Competence in Information-Seeking Agents
• Reinforcement Learning Foundations for Deep Research Systems: A Survey
• Open Data Synthesis For Deep Research
• Deep Research: A Survey of Autonomous Research Agents
• Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models
• MetaAgent: Toward Self-Evolving Agent via Tool Meta-Learning
• From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs

Related

• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
• From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
• Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation