Bounding Hallucinations: Information-Theoretic Guarantees for RAG Systems via Merlin-Arthur Protocols

2025/12/12 by Deiseroth, Björn, Höth, Max Henning, Kersting, Kristian +1
Computer Science · Decision Sciences · Materials Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.10 #I.2.7 #Machine Learning (cs.LG) #Machine Learning in Materials Science #Scientific Computing and Data Management #Topic Modeling

paper · doi:10.48550/arxiv.2512.11614

openalex publication_date 2025/12/12 · openalex created_date 2025/12/16 · openalex updated_date 2026/07/28

Abstract

Retrieval-augmented generation (RAG) relies on retrieved context to guide large language models (LLM), yet treats retrieval as a weak heuristic rather than verifiable evidence -- leading to unsupported answers, hallucinations, and reliance on spurious context. We introduce a novel training framework that treats the RAG pipeline as an interactive proof system by adapting the Merlin-Arthur (M/A) protocol: Arthur (the generator LLM) trains on questions with unknown context provenance and Merlin gives helpful evidence, while Morgana injects adversarial, misleading context. Both use an XAI method to identify and modify evidence most influential to Arthur. This trains Arthur to (1) answer when evidence supports the answer, (2) reject when evidence is insufficient, and (3) rely on the context spans that truly ground the answer. We further introduce a verification framework that disentangles explanation fidelity from model predictive errors, and introduce the Explained Information Fraction (EIF), which normalizes M/A mutual-information guarantees. Across three RAG datasets and multiple LLM families and sizes, M/A training makes LLMs more grounded in evidence, increases information theoretic measures (soundness, completeness) and reject behavior with less hallucinations, without manually annotated unanswerable samples. Finally, the retriever also improves recall and MRR via automatically generated M/A hard positives and negatives. While high accuracy does not guarantee entropy flow from context to answer, our EIF results show that autonomous interactive-proof-style supervision enables RAG systems that treat retrieved documents as verifiable evidence. % rather than suggestions.

Citations

• RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models
• Detecting (Un)answerability in Large Language Models with Linear Directions
• RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems
• Retrieval-Augmented Generation with Conflicting Evidence
• Out of Style: RAG's Fragility to Linguistic Variation
• Tricking Retrievers with Influential Tokens: An Efficient Black-Box Corpus Poisoning Attack
• ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability
• The Llama 3 Herd of Models
• Know Your Limits: A Survey of Abstention in Large Language Models
• Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models
• Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations
• Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback
• RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning
• AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
• Interpretability Guarantees with Merlin-Arthur Classifiers
• Unsupervised Dense Information Retrieval with Contrastive Learning
• Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models
• Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers
• Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
• Explaining NLP Models via Minimal Contrastive Editing (MiCE)
• Dense Passage Retrieval for Open-Domain Question Answering
• Universal Adversarial Triggers for Attacking and Analyzing NLP
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Adversarial Examples for Evaluating Reading Comprehension Systems
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• Axiomatic Attribution for Deep Networks
• Understanding Neural Networks through Representation Erasure
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Rationalizing Neural Predictions

Related

• HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation
• ConfRAG: Confidence-Guided Retrieval-Augmenting Generation
• Stemming Hallucination in Language Models Using a Licensing Oracle
• The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systems
• Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases