ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph

2023/12/30 by Jinhao Jiang, Kun Zhou, Jiang, Jinhao +7 · 7 citations
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2401.00158

openalex publication_date 2023/12/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Question Answering over Knowledge Graph (KGQA) aims to seek answer entities for the natural language question from a large-scale Knowledge Graph~(KG). To better perform reasoning on KG, recent work typically adopts a pre-trained language model~(PLM) to model the question, and a graph neural network~(GNN) based module to perform multi-hop reasoning on the KG. Despite the effectiveness, due to the divergence in model architecture, the PLM and GNN are not closely integrated, limiting the knowledge sharing and fine-grained feature interactions. To solve it, we aim to simplify the above two-module approach, and develop a more capable PLM that can directly support subgraph reasoning for KGQA, namely ReasoningLM. In our approach, we propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning, and also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions. After adaptation, the PLM can be parameter-efficient fine-tuned on downstream tasks. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data. Our codes and data are publicly available at~\urlhttps://github.com/RUCAIBox/ReasoningLM.

Cited by

• ProgRAG: Hallucination-Resistant Progressive Retrieval and Reasoning over Knowledge Graphs
• Self-Correction Distillation for Structured Data Question Answering
• RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs
• Temporal Reasoning with Large Language Models Augmented by Evolving Knowledge Graphs
• VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft
• EEG-MedRAG: Enhancing EEG-based Clinical Decision-Making via Hierarchical Hypergraph Retrieval-Augmented Generation
• DAMR: Efficient and Adaptive Context-Aware Knowledge Graph Question Answering with LLM-Guided MCTS

Related

• SemPool: Simple, robust, and interpretable KG pooling for enhancing language models
• Debate on Graph: a Flexible and Reliable Reasoning Framework for Large Language Models
• ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphs
• GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
• GNN2R: Weakly-Supervised Rationale-Providing Question Answering over Knowledge Graphs