OAEI-LLM: A Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching

2024/09/21 by Zhangcheng Qiang, Qiang, Zhangcheng, Kerry Taylor +5 · 1 citation
Computer Science · Physics and Astronomy · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Complex Network Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2409.14038

openalex publication_date 2024/09/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Hallucinations of large language models (LLMs) commonly occur in domain-specific downstream tasks, with no exception in ontology matching (OM). The prevalence of using LLMs for OM raises the need for benchmarks to better understand LLM hallucinations. The OAEI-LLM dataset is an extended version of the Ontology Alignment Evaluation Initiative (OAEI) datasets that evaluate LLM-specific hallucinations in OM tasks. We outline the methodology used in dataset construction and schema extension, and provide examples of potential use cases.

Cited by

• KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models

Related

• OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching
• LLMs4OM: Matching Ontologies with Large Language Models
• HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild
• A comprehensive taxonomy of hallucinations in Large Language Models
• Improving LLM-based Ontology Matching with fine-tuning on synthetic data