COMPKE: Complex Question Answering under Knowledge Editing

2025/06/01 by Keyuan Cheng, Cheng, Keyuan, Zhou He +11 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.00829

openalex publication_date 2025/06/01 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Knowledge Editing, which efficiently modifies the knowledge in large language models, has gathered great attention. Current benchmarks primarily use multi-hop question answering to assess and analyze newly injected or updated knowledge. However, we argue that these benchmarks fail to effectively evaluate how well the updated models apply this knowledge in real-life scenarios, particularly when questions require complex reasoning, involving one-to-many relationships or multi-step logical intersections. To fill in this gap, we introduce a new benchmark, COMPKE: Complex Question Answering under Knowledge Editing, which includes 11,924 complex questions that reflect real-life situations. We conduct an extensive evaluation of four knowledge editing methods on COMPKE, revealing that their effectiveness varies notably across different models. For instance, MeLLo attains an accuracy of 39.47 on GPT-4O-MINI, but this drops sharply to 3.83 on QWEN2.5-3B. We further investigate the underlying causes of these disparities from both methodological and model-specific perspectives. The datasets are available at https://github.com/kzjkzj666/CompKE.

Citations

• Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planning
• Understanding the Repeat Curse in Large Language Models from a Feature Perspective
• Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements
• Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning
• EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification
• Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing
• Can Knowledge Editing Really Correct Hallucinations?
• Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing
• MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language
• The Llama 3 Herd of Models
• Knowledge Mechanisms in Large Language Models: A Survey and Perspective
• Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?
• A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning
• Leveraging Logical Rules in Knowledge Editing: A Cherry on the Top
• What does the Knowledge Neuron Thesis Have to do with Knowledge?
• Multi-hop Question Answering under Temporal Knowledge Editing
• Editing Knowledge Representation of Language Model via Rephrased Prefix Prompts
• CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledge
• Knowledge Graph Enhanced Large Language Model Editing
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse
• Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks
• Model Editing at Scale leads to Gradual and Catastrophic Forgetting
• Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
• PokeMQA: Programmable knowledge editing for Multi-hop Question Answering
• Evaluating the Reversal Curse in Model Editing
• Evaluating the Ripple Effects of Knowledge Editing in Language Models
• MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
• QLoRA: Efficient Finetuning of Quantized LLMs
• DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text
• Can We Edit Factual Knowledge by In-Context Learning?
• Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge
• GPT-4 Technical Report
• Reinforcement Learning-based Counter-Misinformation Response Generation: A Case Study of COVID-19 Vaccine Misinformation
• Transformer-Patcher: One Mistake worth One Neuron
• Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
• Mass-Editing Memory in a Transformer
• Memory-Based Model Editing at Scale
• Locating and Editing Factual Associations in GPT
• LoRA: Low-Rank Adaptation of Large Language Models
• Fine-Grained Named Entity Typing over Distantly Supervised Data Based on Refined Representations
• The Web as a Knowledge-base for Answering Complex Questions
• Variational Reasoning for Question Answering with Knowledge Graph
• Zero-Shot Relation Extraction via Reading Comprehension
• Exploring the Personality Traits of LLMs through Latent Features Steering

Cited by

• The Compositional Architecture of Regret in Large Language Models
• Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs
• Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images

Related

• Leveraging Logical Rules in Knowledge Editing: A Cherry on the Top
• PokeMQA: Programmable knowledge editing for Multi-hop Question Answering
• MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
• Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysis
• Avoiding Knowledge Edit Skipping in Multi-hop Question Answering with Guided Decomposition