KEO: Knowledge Extraction on OMIn via Knowledge Graphs and RAG for Safety-Critical Aviation Maintenance

2025/10/07 by Kuangshi Ai, Jonathan A. Karr, Ai, Kuangshi +7
Decision Sciences · #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Risk and Safety Analysis

paper · pdf · doi:10.48550/arxiv.2510.05524

openalex publication_date 2025/10/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Knowledge Extraction on OMIn (KEO), a domain-specific knowledge extraction and reasoning framework with large language models (LLMs) in safety-critical contexts. Using the Operations and Maintenance Intelligence (OMIn) dataset, we construct a QA benchmark spanning global sensemaking and actionable maintenance tasks. KEO builds a structured Knowledge Graph (KG) and integrates it into a retrieval-augmented generation (RAG) pipeline, enabling more coherent, dataset-wide reasoning than traditional text-chunk RAG. We evaluate locally deployable LLMs (Gemma-3, Phi-4, Mistral-Nemo) and employ stronger models (GPT-4o, Llama-3.3) as judges. Experiments show that KEO markedly improves global sensemaking by revealing patterns and system-level insights, while text-chunk RAG remains effective for fine-grained procedural tasks requiring localized retrieval. These findings underscore the promise of KG-augmented LLMs for secure, domain-specific QA and their potential in high-stakes reasoning. The code is available at https://github.com/JonathanKarr33/keo.

Citations

• SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
• An Evaluation-Centric Paradigm for Scientific Visualization Agents
• CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance
• Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
• Trusted Knowledge Extraction for Operations and Maintenance Intelligence
• TexGS-VolVis: Expressive Scene Editing for Volume Visualization via Textured Gaussian Splatting
• LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
• Knowledge Graph-Guided Retrieval Augmented Generation
• MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain
• A Survey on LLM-as-a-Judge
• Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates
• Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• Benchmarking Retrieval-Augmented Generation for Medicine
• MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
• Artificial Intelligence for Safety-Critical Systems in Industrial and Transportation Domains: A Survey
• On the Evolution of Knowledge Graphs: A Survey and Perspective
• Knowledge Graph Prompting for Multi-Document Question Answering
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
• Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study
• In-Context Retrieval-Augmented Language Models
• Improving language models by retrieving from trillions of tokens
• MaintNet: A Collaborative Open-Source Library for Predictive Maintenance Language Resources
• Entities as Experts: Sparse Memory Access with Entity Supervision
• Knowledge Graphs
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Integrating Large Language Model for Improved Causal Discovery
• Mention Memory: incorporating textual knowledge into Transformers through entity mention attention

Related

• Trusted Knowledge Extraction for Operations and Maintenance Intelligence
• Knowledge Graph - Deep Learning: A Case Study in Question Answering in Aviation Safety Domain
• NOTAM-Evolve: A Knowledge-Guided Self-Evolving Optimization Framework with LLMs for NOTAM Interpretation
• CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance
• KG-RAG: Bridging the Gap Between Knowledge and Creativity