2025/10/17 by Yi Wang, Jiuqi Wang, Wan, Yi +9
Computer Science · Materials Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #FOS: Computer and information sciences #Machine Learning in Materials Science #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2510.15862
openalex publication_date 2025/10/17 · openalex created_date 2025/10/21 · openalex updated_date 2026/07/31
Large language models (LLMs) augmented with external tools are increasingly deployed as deep research agents that gather, reason over, and synthesize web information to answer complex queries. Although recent open-source systems achieve strong empirical performance via reinforcement learning from web interactions, the impact of key design choices remains under-explored. We formalize deep research as reinforcement learning in an episodic finite Markov decision process and construct a competitive baseline agent grounded in this formulation. Building on this foundation, we systematically examine critical design decisions at both training and inference time and identify four factors that substantially improve performance: replacing rule-based rewards with AI feedback from an LLM judge, fine-tuning with the on-policy RLOO algorithm instead of the off-policy GRPO algorithm, filtering low-quality training samples, and employing an error-tolerant test-time rollout strategy. Together, these design choices yield a deep research agent that establishes state-of-the-art performance among 7B-scale agents when evaluated across ten widely used benchmarks.