LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

2024/10/03 by Hadas Orgad, Orgad, Hadas, Michael Toker +11 · 10 voices · 52 citations
Computer Science · Economics, Econometrics and Finance · Business, Management and Accounting · #Blockchain Technology Applications and Security #Credit Risk and Financial Regulations #Corporate Insolvency and Governance

paper · pdf · doi:10.48550/arxiv.2410.02707

Abstract

Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that -- contrary to prior claims -- truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

Cited by

• Dissociating the Internal Representations of Sycophancy in LLMs
• Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
• Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade
• Reliability Scales Inversely: Hallucinations Snowball Faster in Bigger Language Models
• Hallucinations Undermine Trust; Metacognition is a Way Forward
• Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debate
• Convergent Evolution: How Different Language Models Learn Similar Number Representations
• Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
• Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testing
• D-Score: A Spectral Hidden-State Signal for Hallucination Detection in Large Language Models
• xMIx: High-Performance Serving-Time Platform for Mechanistic Interpretability Apps
• HeadHunt-VAD: Hunting Robust Anomaly-Sensitive Heads in MLLM for Tuning-Free Video Anomaly Detection
• Inflation Attitudes of Large Language Models
• Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes
• Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs
• Epistemic Fragility in Large Language Models: Prompt Framing Systematically Modulates Misinformation Correction
• The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguity
• How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
• RepV: Safety-Separable Latent Spaces for Scalable Neurosymbolic Plan Verification
• Attention-Aligned Reasoning for Large Language Models
• Same Evidence, Different Target: Decoding How Diagnostic Evidence Bears on Causal Questions from Language-Model States
• HACK: Hallucinations Along Certainty and Knowledge Axes
• Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration
• Mixture-of-Minds: Multi-Agent Reinforcement Learning for Table Understanding
• CARES: Context-Aware Resolution Selector for VLMs
• Emergence of Linear Truth Encodings in Language Models
• Stable but Miscalibrated: A Kantian View on Overconfidence from Filters to Large Language Models
• LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance
• Therefore I am. I Think
• Large Language Models Do NOT Really Know What They Don't Know
• The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs
• "Sometimes You Need Facts, and Sometimes a Hug": Understanding Older Adults' Preferences for Explanations in LLM-Based Conversational AI Systems
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models
• LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization
• Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViT
• TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
• Neural Message-Passing on Attention Graphs for Hallucination Detection
• Reference-Free Rating of LLM Responses via Latent Information
• Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions
• Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
• The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models
• Estimating Semantic Alphabet Size for LLM Uncertainty Quantification
• When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment
• Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detection
• Beyond Transcription: Mechanistic Interpretability in ASR
• Trustworthy Agents for Electronic Health Records through Confidence Estimation
• Answering the Unanswerable Is to Err Knowingly: Analyzing and Mitigating Abstention Failures in Large Reasoning Models
• Real-Time Detection of Hallucinated Entities in Long-Form Generation
• Prompt-Induced Linguistic Fingerprints for LLM-Generated Fake News Detection
• Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models
• Introducing Sustainable Publishing
• The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs

Discussions

• LLMs know more than they show: On the intrinsic representation of hallucinations [hn, 137 points, 140 comments]
• When LLMs make errors (“hallucinate”), their internal states often contain clues about the truthfulness of the output. This paper shows those clues are concentrated in specific tokens. arxiv.org/abs/2 [bsky, 54 points, 3 comments]
• Very happy to have our paper accepted at #ICLR2025: ➡️ LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations arxiv.org/abs/2410.02707 [bsky, 27 points, 0 comments]
• LLMs Know More Than They Show: arxiv.org/abs/2410.02707 * Adding a true-seeking classifier probe on the token embeddings can have better performance than the actual generation * Is something wrong g [bsky, 8 points, 0 comments]
• arxiv.org/abs/2410.02707 🧪Hallucinations in Generative AI/Large Language Models - cues re truthfulness lie within …. [bsky, 3 points, 1 comments]
• Internal representations of LLMs encode information about truthfulness https://arxiv.org/abs/2410.02707 https://news.ycombinator.com/item?id=41995201 [bsky, 1 points, 0 comments]
• 🧠 LLMs are hiding secrets! A new study reveals they *know* when they’re wrong—but don’t show it. Hidden “truth tokens” could help detect AI errors & fix hallucinations. Are we underestimating their s [bsky, 1 points, 0 comments]
• Internal representations of LLMs encode information about truthfulness [bsky, 0 points, 0 comments]
• Internal representations of LLMs encode information about truthfulness https://arxiv.org/abs/2410.02707 [bsky, 0 points, 1 comments]
• This research reveals that LLMs actually encode more information about truthfulness than previously thought! This can be used to detect errors. Also highlights the potential to predict types of errors [bsky, 0 points, 0 comments]

Related

• On the Universal Truthfulness Hyperplane Inside LLMs
• Large Language Models Do NOT Really Know What They Don't Know
• Distinguishing Ignorance from Error in LLM Hallucinations
• Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
• A Debate-Driven Experiment on LLM Hallucinations and Accuracy