KG-LLM-Bench: A Scalable Benchmark for Evaluating LLM Reasoning on Textualized Knowledge Graphs

2025/04/09 by Elan Markowitz, Krupa Galiya, Markowitz, Elan +5 · 1 citation
Computer Science · #Advanced Graph Neural Networks #Topic Modeling #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2504.07087

Abstract

Knowledge graphs have emerged as a popular method for injecting up-to-date, factual knowledge into large language models (LLMs). This is typically achieved by converting the knowledge graph into text that the LLM can process in context. While multiple methods of encoding knowledge graphs have been proposed, the impact of this textualization process on LLM performance remains under-explored. We introduce KG-LLM-Bench, a comprehensive and extensible benchmark spanning five knowledge graph understanding tasks, and evaluate how different encoding strategies affect performance across various base models. Our extensive experiments with seven language models and five textualization strategies provide insights for optimizing LLM performance on KG reasoning tasks.

Cited by

• Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reports

Related

• Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Data
• Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering
• Large Language Models Can Better Understand Knowledge Graphs Than We Thought
• KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models
• LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities