Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer

2025/02/18 by Adi Simhi, Itay Itzhak, Simhi, Adi +7 · 1 voice · 16 citations
Computer Science · Neuroscience · #Blockchain Technology Applications and Security #Hallucinations in medical conditions #cs.CL

paper · pdf · doi:10.48550/arxiv.2502.12964

arxiv published 2025/02/18 · arxiv updated 2025/08/25

Abstract

Prior work on large language model (LLM) hallucinations has associated them with model uncertainty or inaccurate knowledge. In this work, we define and investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation, which can happen in real-world settings, causes it to produce a hallucinated response with high certainty. This phenomenon, which we dub CHOKE (Certain Hallucinations Overriding Known Evidence), is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability. We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations. This difference leads existing mitigation methods to perform worse on CHOKE examples than on general hallucinations. Finally, we introduce a probing-based mitigation that outperforms existing methods on CHOKE hallucinations. These findings reveal an overlooked aspect of hallucinations, emphasizing the need to understand their origins and improve mitigation strategies to enhance LLM safety. The code is available at https://github.com/technion-cs-nlp/Trust_me_Im_wrong .

Cited by

• SPAD: Seven-Source Token Probability Attribution with Syntactic Aggregation for Detecting Hallucinations in RAG
• PEFA-AI: Advancing Open-source LLMs for RTL generation using Progressive Error Feedback Agentic-AI
• DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching
• BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence
• Large Language Models Hallucination: A Comprehensive Survey
• Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting
• Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems
• Leveraging NTPs for Efficient Hallucination Detection in VLMs
• FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMs
• Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD
• Tractable Asymmetric Verification for Large Language Models via Deterministic Replicability
• Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate
• A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinations
• Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges
• The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systems
• HIDE and Seek: Detecting Hallucinations in Language Models via Decoupled Representations

Discussions

• 🔍Check out our paper "Trust Me, I’m Wrong: High-Certainty Hallucinations in LLMs", at arxiv.org/pdf/2502.12964 and code at github.com/technion-cs-... [bsky, 3 points, 0 comments]

Related

• Distinguishing Ignorance from Error in LLM Hallucinations
• On Large Language Models' Hallucination with Regard to Known Facts
• Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States
• HACK: Hallucinations Along Certainty and Knowledge Axes
• LLMs Will Always Hallucinate, and We Need to Live With This