ESI: Epistemic Uncertainty Quantification via Semantic-preserving Intervention for Large Language Models

2025/10/15 by Mingda Li, Li, Mingda, Xinyu Li +5
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.13103

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Uncertainty Quantification (UQ) is a promising approach to improve model reliability, yet quantifying the uncertainty of Large Language Models (LLMs) is non-trivial. In this work, we establish a connection between the uncertainty of LLMs and their invariance under semantic-preserving intervention from a causal perspective. Building on this foundation, we propose a novel grey-box uncertainty quantification method that measures the variation in model outputs before and after the semantic-preserving intervention. Through theoretical justification, we show that our method provides an effective estimate of epistemic uncertainty. Our extensive experiments, conducted across various LLMs and a variety of question-answering (QA) datasets, demonstrate that our method excels not only in terms of effectiveness but also in computational efficiency.

Citations

• A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions
• Causal-Guided Active Learning for Debiasing Large Language Models
• To Believe or Not to Believe Your LLM
• Language Models with Conformal Factuality Guarantees
• INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling
• Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• 🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Uncertainty in Natural Language Generation: From Theory to Applications
• A Comprehensive Overview of Large Language Models
• Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
• Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
• Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• Evaluating Open-Domain Question Answering in the Era of Large Language Models
• Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
• Language Models (Mostly) Know What They Know
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
• Locally Typical Sampling
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• A Survey of Uncertainty in Deep Neural Networks
• On Hallucination and Predictive Uncertainty in Conditional Language Generation
• Reducing conversational agents' overconfidence through linguistic calibration
• AmbigQA: Answering Ambiguous Open-domain Questions
• Shortcut learning in deep neural networks
• Invariant Risk Minimization
• The Curious Case of Neural Text Degeneration
• CoQA: A Conversational Question Answering Challenge
• Crowdsourcing Multiple Choice Science Questions
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension

Related

• Improving Uncertainty Quantification in Large Language Models via Semantic Embeddings
• Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey
• Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuning
• SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models
• Measuring Aleatoric and Epistemic Uncertainty in LLMs: Empirical Evaluation on ID and OOD QA Tasks