AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction

2025/10/17 by Tsang, Hong Ting, Bai, Jiaxin, Huang, Haoyu +5
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.15339

Abstract

Building effective knowledge graphs (KGs) for Retrieval-Augmented Generation (RAG) is pivotal for advancing question answering (QA) systems. However, its effectiveness is hindered by a fundamental disconnect: the knowledge graph (KG) construction process is decoupled from its downstream application, yielding suboptimal graph structures. To bridge this gap, we introduce AutoGraph-R1, the first framework to directly optimize KG construction for task performance using Reinforcement Learning (RL). AutoGraph-R1 trains an LLM constructor by framing graph generation as a policy learning problem, where the reward is derived from the graph's functional utility in a RAG pipeline. We design two novel, task-aware reward functions, one for graphs as knowledge carriers and another as knowledge indices. Across multiple QA benchmarks, AutoGraph-R1 consistently enables graph RAG methods to achieve significant performance gains over using task-agnostic baseline graphs. Our work shows it is possible to close the loop between construction and application, shifting the paradigm from building intrinsically ``good'' graphs to building demonstrably ``useful'' ones.

Citations

• AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM
• A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges
• Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning
• Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
• When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation
• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
• AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corpora
• Reason-Align-Respond: Aligning LLM Reasoning with Knowledge Graphs for KGQA
• ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning
• s3: You Don't Need That Much Data to Train a Search Agent via RL
• From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
• Understanding R1-Zero-Like Training: A Critical Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning
• From RAG to Memory: Non-Parametric Continual Learning for Large Language Models
• RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
• KGGen: Extracting Knowledge Graphs from Plain Text with Language Models
• Top Ten Challenges Towards Agentic Neural Graph Databases
• A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models
• Qwen2.5 Technical Report
• Can LLMs be Good Graph Judge for Knowledge Graph Construction?
• Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation
• RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs
• AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Data
• StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization
• Neuro-Symbolic Entity Alignment via Variational Inference
• iText2KG: Incremental Knowledge Graphs Construction Using Large Language Models
• Graph Retrieval-Augmented Generation: A Survey
• Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation
• CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
• Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation
• GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models
• Knowledge Graph-Enhanced Large Language Models via Path Selection
• GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model
• Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning
• HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
• Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Advancing Abductive Reasoning in Knowledge Graphs through Complex Logical Hypothesis Generation
• A Survey of Reinforcement Learning from Human Feedback
• Retrieval-Augmented Generation for Large Language Models: A Survey
• KnowGPT: Knowledge Graph based Prompting for Large Language Models
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis
• Knowledge Graph Prompting for Multi-Document Question Answering
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
• PiVe: Prompting with Iterative Verification Improving Graph-based Generative Capability of LLMs
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
• Atlas: Few-shot Learning with Retrieval Augmented Language Models
• Training language models to follow instructions with human feedback
• Survey of Hallucination in Natural Language Generation
• ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models
• MuSiQue: Multihop Questions via Single-hop Question Composition
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Proximal Policy Optimization Algorithms

Related

• Efficient and Transferable Agentic Knowledge Graph RAG via Reinforcement Learning
• Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs
• Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning
• GRAIL:Learning to Interact with Large Knowledge Graphs for Retrieval Augmented Reasoning
• RouteRAG: Efficient Retrieval-Augmented Generation from Text and Graph via Reinforcement Learning