RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs

2025/09/25 by Zhengwang Jiang, Lin, Can, Jiang, Zhengwang +9 · 1 citation
Computer Science · Decision Sciences · #Semantic Web and Ontologies #Data Quality and Management #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.01257

Abstract

Knowledge graph question answering (KGQA) aims to answer natural language questions using knowledge graphs. Recent research leverages large language models (LLMs) to enhance KGQA reasoning, but faces limitations: retrieval-based methods are constrained by the quality of retrieved information, while agent-based methods rely heavily on proprietary LLMs. To address these limitations, we propose Retrieval-Judgment-Exploration (RJE), a framework that retrieves refined reasoning paths, evaluates their sufficiency, and conditionally explores additional evidence. Moreover, RJE introduces specialized auxiliary modules enabling small-sized LLMs to perform effectively: Reasoning Path Ranking, Question Decomposition, and Retriever-assisted Exploration. Experiments show that our approach with proprietary LLMs (such as GPT-4o-mini) outperforms existing baselines while enabling small open-source LLMs (such as 3B and 8B parameters) to achieve competitive results without fine-tuning LLMs. Additionally, RJE substantially reduces the number of LLM calls and token usage compared to agent-based methods, yielding significant efficiency improvements.

Citations

• Reasoning of Large Language Models over Knowledge Graphs with Super-Relations
• Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs
• Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains
• Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs
• GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning
• KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graph
• Enhancing Complex Question Answering over Knowledge Graphs through Evidence Pattern Retrieval
• ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
• FC-KBQA: A Fine-to-Coarse Composition Framework for Knowledge Base Question Answering
• Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
• Graph Reasoning for Question Answering with Triplet Retrieval
• Direct Fact Retrieval from Knowledge Graphs without Entity Linking
• StructGPT: A General Framework for Large Language Model to Reason over Structured Data
• UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Program Transfer for Answering Complex Questions over Knowledge Bases
• Language Models are Few-Shot Learners
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• The Web as a Knowledge-base for Answering Complex Questions

Cited by

• GraphIF: Enhancing Multi-Turn Instruction Following for Large Language Models with Relation Graph Prompt

Related

• Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering
• Enhancing Large Language Models with Reward-guided Tree Search for Knowledge Graph Question and Answering
• FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering
• KGFR: A Foundation Retriever for Generalized Knowledge Graph Question Answering
• Knowledge Graph-extended Retrieval Augmented Generation for Question Answering