LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

2024/10/03 by Hadas Orgad, Orgad, Hadas, Michael Toker +11 · 10 voices · 104 citations
Business, Management and Accounting · Computer Science · Economics, Econometrics and Finance · Psychology · #Blockchain Technology Applications and Security #Cognitive psychology #Corporate Insolvency and Governance #Credit Risk and Financial Regulations #Law #Political science #Politics #Psychology #Representation (politics) #Social psychology

paper · pdf · doi:10.48550/arxiv.2410.02707

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/10/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that -- contrary to prior claims -- truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

Cited by

• Dissociating the Internal Representations of Sycophancy in LLMs
• Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
• Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade
• Reliability Scales Inversely: Hallucinations Snowball Faster in Bigger Language Models
• Hallucinations Undermine Trust; Metacognition is a Way Forward
• Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debate
• Convergent Evolution: How Different Language Models Learn Similar Number Representations
• Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
• Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testing
• D-Score: A Spectral Hidden-State Signal for Hallucination Detection in Large Language Models
• xMIx: High-Performance Serving-Time Platform for Mechanistic Interpretability Apps
• HeadHunt-VAD: Hunting Robust Anomaly-Sensitive Heads in MLLM for Tuning-Free Video Anomaly Detection
• Inflation Attitudes of Large Language Models
• Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes
• Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs
• Epistemic Fragility in Large Language Models: Prompt Framing Systematically Modulates Misinformation Correction
• The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguity
• How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
• RepV: Safety-Separable Latent Spaces for Scalable Neurosymbolic Plan Verification
• Attention-Aligned Reasoning for Large Language Models
• Same Evidence, Different Target: Decoding How Diagnostic Evidence Bears on Causal Questions from Language-Model States
• HACK: Hallucinations Along Certainty and Knowledge Axes
• Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration
• Mixture-of-Minds: Multi-Agent Reinforcement Learning for Table Understanding
• CARES: Context-Aware Resolution Selector for VLMs
• Emergence of Linear Truth Encodings in Language Models
• Stable but Miscalibrated: A Kantian View on Overconfidence from Filters to Large Language Models
• LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance
• Therefore I am. I Think
• Large Language Models Do NOT Really Know What They Don't Know
• The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs
• "Sometimes You Need Facts, and Sometimes a Hug": Understanding Older Adults' Preferences for Explanations in LLM-Based Conversational AI Systems
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models
• LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization
• Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViT
• TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
• Neural Message-Passing on Attention Graphs for Hallucination Detection
• Reference-Free Rating of LLM Responses via Latent Information
• Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions
• Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
• The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models
• Estimating Semantic Alphabet Size for LLM Uncertainty Quantification
• When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment
• Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detection
• Beyond Transcription: Mechanistic Interpretability in ASR
• Trustworthy Agents for Electronic Health Records through Confidence Estimation
• Answering the Unanswerable Is to Err Knowingly: Analyzing and Mitigating Abstention Failures in Large Reasoning Models
• Real-Time Detection of Hallucinated Entities in Long-Form Generation
• Prompt-Induced Linguistic Fingerprints for LLM-Generated Fake News Detection
• Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models
• Introducing Sustainable Publishing
• ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
• The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs
• Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
• HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs
• Probing for Arithmetic Errors in Language Models
• Large Language Models Encode Semantics and Alignment in Linearly Separable Representations
• The Geometries of Truth Are Orthogonal Across Tasks
• The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality
• Reconstructing Biological Pathways by Applying Selective Incremental Learning to (Very) Small Language Models
• Toward Better Generalisation in Uncertainty Estimators: Leveraging Data-Agnostic Features
• Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trust
• Persona Features Control Emergent Misalignment
• Hallucination Detection with Small Language Models
• Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
• CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection
• Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety
• One SPACE to Rule Them All: Jointly Mitigating Factuality and Faithfulness Hallucinations in LLMs
• Growing Through Experience: Scaling Episodic Grounding in Language Models
• Probing Neural Topology of Large Language Models
• HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMs
• How Does Response Length Affect Long-Form Factuality
• Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMs
• Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generation
• Whose Name Comes Up? Auditing LLM-Based Scholar Recommendations
• Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs
• When Do LLMs Admit Their Mistakes? Understanding The Role Of Model Belief In Retraction
• HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving
• RePPL: Recalibrating Perplexity by Uncertainty in Semantic Propagation and Language Generation for Explainable QA Hallucination Detection
• Traveling Across Languages: Benchmarking Cross-Lingual Consistency in Multimodal LLMs
• Void in Language Models
• Truth Neurons
• BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMs
• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors
• MTRE: Multi-Token Reliability Estimation for Hallucination Detection in VLMs
• Revealing economic facts: LLMs know more than they say
• C-MIG: Multi-view Information Gain-based Retrieval-Augmented Generation for Clinical Diagnosis Reasoning
• Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
• Your Agentic LLMs Secretly Encode Latent Signals of Indirect Prompt-Injection Exposure
• Reading Between the Tokens: Improving Preference Predictions through Mechanistic Forecasting
• Spilled Energy in Large Language Models
• Interpretability Can Be Actionable
• The Truthfulness Spectrum Hypothesis
• Investigating task-specific prompts and sparse autoencoders for activation monitoring
• Reasoning Models Know What's Important, and Encode It in Their Activations
• Surgical Repair of Insecure Code Generation in LLMs
• Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls
• Language Models Encode the Contextual Truth of Propositions
• The Geometry of Self-Verification in a Task-Specific Reasoning Model
• TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs
• To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling
• Hallucination Detection in LLMs with Topological Divergence on Attention Graphs
• Robust Hallucination Detection in LLMs via Adaptive Token Selection

Discussions

• LLMs know more than they show: On the intrinsic representation of hallucinations [hn, 137 points, 140 comments]
• When LLMs make errors (“hallucinate”), their internal states often contain clues about the truthfulness of the output. This paper shows those clues are concentrated in specific tokens. arxiv.org/abs/2 [bsky, 54 points, 3 comments]
• Very happy to have our paper accepted at #ICLR2025: ➡️ LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations arxiv.org/abs/2410.02707 [bsky, 27 points, 0 comments]
• LLMs Know More Than They Show: arxiv.org/abs/2410.02707 * Adding a true-seeking classifier probe on the token embeddings can have better performance than the actual generation * Is something wrong g [bsky, 8 points, 0 comments]
• arxiv.org/abs/2410.02707 🧪Hallucinations in Generative AI/Large Language Models - cues re truthfulness lie within …. [bsky, 3 points, 1 comments]
• Internal representations of LLMs encode information about truthfulness https://arxiv.org/abs/2410.02707 https://news.ycombinator.com/item?id=41995201 [bsky, 1 points, 0 comments]
• 🧠 LLMs are hiding secrets! A new study reveals they *know* when they’re wrong—but don’t show it. Hidden “truth tokens” could help detect AI errors & fix hallucinations. Are we underestimating their s [bsky, 1 points, 0 comments]
• Internal representations of LLMs encode information about truthfulness [bsky, 0 points, 0 comments]
• Internal representations of LLMs encode information about truthfulness https://arxiv.org/abs/2410.02707 [bsky, 0 points, 1 comments]
• This research reveals that LLMs actually encode more information about truthfulness than previously thought! This can be used to detect errors. Also highlights the potential to predict types of errors [bsky, 0 points, 0 comments]

Related

• On the Universal Truthfulness Hyperplane Inside LLMs
• Large Language Models Do NOT Really Know What They Don't Know
• Distinguishing Ignorance from Error in LLM Hallucinations
• Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
• A Debate-Driven Experiment on LLM Hallucinations and Accuracy