Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations

2025/06/03 by Jinyuan Luo, Luo, Jinyuan, Zhen Fang +6
Medicine · Neuroscience · #Artificial Intelligence (cs.AI) #Autoimmune Neurological Disorders and Treatments #Bayesian probability #Drug-Induced Ocular Toxicity #FOS: Computer and information sciences #Hallucinating #Hallucinations in medical conditions #Inference #Key (lock) #Metric (unit) #Noise (video) #Prior probability #Sensitivity (control systems)

paper · pdf · doi:10.48550/arxiv.2506.02696

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/03 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Hallucination remains a key obstacle to the reliable deployment of large language models (LLMs) in real-world question answering tasks. A widely adopted strategy to detect hallucination, known as self-assessment, relies on the model's own output confidence to estimate the factual accuracy of its answers. However, this strategy assumes that the model's output distribution closely reflects the true data distribution, which may not always hold in practice. As bias accumulates through the model's layers, the final output can diverge from the underlying reasoning process, making output-level confidence an unreliable signal for hallucination detection. In this work, we propose Sample-Specific Prompting (SSP), a new framework that improves self-assessment by analyzing perturbation sensitivity at intermediate representations. These representations, being less influenced by model bias, offer a more faithful view of the model's latent reasoning process. Specifically, SSP dynamically generates noise prompts for each input and employs a lightweight encoder to amplify the changes in representations caused by the perturbation. A contrastive distance metric is then used to quantify these differences and separate truthful from hallucinated responses. By leveraging the dynamic behavior of intermediate representations under perturbation, SSP enables more reliable self-assessment. Extensive experiments demonstrate that SSP significantly outperforms prior methods across a range of hallucination detection benchmarks.

Citations

• Attack and defense techniques in large language models: A survey and new perspectives
• HalluShift: Measuring Distribution Shifts towards Hallucination Detection in LLMs
• Steer LLM Latents for Hallucination Detection
• Do Large Language Models Know How Much They Know?
• Qwen2.5 Technical Report
• HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection
• The Llama 3 Herd of Models
• Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps
• Sample-specific Masks for Visual Reprogramming-based Prompting
• Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models
• Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension
• Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States
• INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
• A Survey on Hallucination in Large Vision-Language Models
• Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation
• Hallucination is Inevitable: An Innate Limitation of Large Language Models
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• Alleviating Hallucinations of Large Language Models through Induced Hallucinations
• On Early Detection of Hallucinations in Factual Question Answering
• Self-Evaluation Improves Selective Generation in Large Language Models
• Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
• Do Language Models Know When They're Hallucinating References?
• AlignScore: Evaluating Factual Consistency with a Unified Alignment Function
• Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation
• Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• LM vs LM: Detecting Factual Errors via Cross Examination
• The Internal State of an LLM Knows When It's Lying
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
• GPT-4 Technical Report
• Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
• Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
• Discovering Latent Knowledge in Language Models Without Supervision
• Out-of-Distribution Detection and Selective Generation for Conditional Language Models
• Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation
• Language Models (Mostly) Know What They Know
• Teaching Models to Express Their Uncertainty in Words
• Visual Prompt Tuning
• Locating and Editing Factual Associations in GPT
• Survey of Hallucination in Natural Language Generation
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Gradient-based Adversarial Attacks against Text Transformers
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• BLEURT: Learning Robust Metrics for Text Generation
• TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
• A Primer in BERTology: What we know about how BERT works
• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Universal Adversarial Triggers for Attacking and Analyzing NLP
• BERTScore: Evaluating Text Generation with BERT
• Identifying and Controlling Important Neurons in Neural Machine Translation
• CoQA: A Conversational Question Answering Challenge
• On Calibration of Modern Neural Networks
• Attention Is All You Need
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• Deep Information Propagation
• Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Related

• Cost-Effective Hallucination Detection for LLMs
• On Early Detection of Hallucinations in Factual Question Answering
• HalluShift: Measuring Distribution Shifts towards Hallucination Detection in LLMs
• Counterfactual Probing for Hallucination Detection and Mitigation in Large Language Models
• SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models