Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learning

2025/05/20 by Wu, Yihong, Ma, Liheng, Li, Muzhi +7 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.17086

Abstract

Large Language Models (LLMs) equipped with modern Retrieval-Augmented Generation (RAG) systems often employ multi-turn interaction pipelines to interface with search engines for complex reasoning tasks. However, such multi-turn interactions inevitably produce long intermediate contexts, as context length grows exponentially with exploration depth. This leads to a well-known limitation of LLMs: their difficulty in effectively leveraging information from long contexts. This problem is further amplified in RAG systems that depend on in-context learning, where few-shot demonstrations must also be included in the prompt, compounding the context-length bottleneck. To address these challenges, we propose Mujica-MyGo, a unified framework for efficient multi-turn reasoning in RAG. Inspired by the divide-and-conquer principle, we introduce Mujica (Multi-hop Joint Intelligence for Complex Question Answering), a multi-agent RAG workflow that decomposes multi-turn interactions into cooperative sub-interactions, thereby mitigating long-context issues. To eliminate the dependency on in-context learning, we further develop MyGO (Minimalist Policy Gradient Optimization), a lightweight and efficient reinforcement learning algorithm that enables effective post-training of LLMs within complex RAG pipelines. We provide theoretical guarantees for MyGO's convergence to the optimal policy. Empirical evaluations across diverse question-answering benchmarks, covering both text corpora and knowledge graphs, show that Mujica-MyGO achieves superior performance.

Citations

• ZeroSearch: Incentivize the Search Capability of LLMs without Searching
• A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
• GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
• Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems
• Fast Think-on-Graph: Wider, Deeper and Faster Reasoning of Large Language Model on Knowledge Graph
• LLM-based Discriminative Reasoning for Knowledge Graph Question Answering
• Retrieval, Reasoning, Re-ranking: A Context-Enriched Framework for Knowledge Graph Completion
• Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs
• Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains
• Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning
• Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models
• Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach
• RaFe: Ranking Feedback Improves Query Rewriting for RAG
• FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering
• IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues
• Understanding the performance gap between online and offline alignment algorithms
• Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering
• SGLang: Efficient Execution of Structured Language Model Programs
• CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
• Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
• Lost in the Middle: How Language Models Use Long Contexts
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• Similarity search in the blink of an eye with compressed indices
• How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval
• A Survey on In-context Learning
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases
• Training language models to follow instructions with human feedback
• MuSiQue: Multihop Questions via Single-hop Question Composition
• HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questions
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• Dense Passage Retrieval for Open-Domain Question Answering
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Proximal Policy Optimization Algorithms
• SGDR: Stochastic Gradient Descent with Warm Restarts
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

Cited by

• It Takes Two: Your GRPO Is Secretly DPO
• An Entity Linking Agent for Question Answering

Related

• Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
• SIRAG: Towards Stable and Interpretable RAG with A Process-Supervised Multi-Agent Framework
• RAG-R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
• Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning
• MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval