How Language Model Hallucinations Can Snowball

2023/05/22 by Muru Zhang, Zhang, Muru, Ofir Press +7 · 35 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.13534

openalex publication_date 2023/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements. Hallucinations are often attributed to knowledge gaps in LMs, but we hypothesize that in some cases, when justifying previously generated hallucinations, LMs output false claims that they can separately recognize as incorrect. We construct three question-answering datasets where ChatGPT and GPT-4 often state an incorrect answer and offer an explanation with at least one incorrect claim. Crucially, we find that ChatGPT and GPT-4 can identify 67% and 87% of their own mistakes, respectively. We refer to this phenomenon as hallucination snowballing: an LM over-commits to early mistakes, leading to more mistakes that it otherwise would not make.

Cited by

• Old Habits Die Hard: How Conversational History Geometrically Traps LLMs
• Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages
• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• Opportunities and Challenges in Harnessing Digital Technology for Effective Teaching and Learning
• Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
• PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise
• When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs
• A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI
• Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems
• HACK: Hallucinations Along Certainty and Knowledge Axes
• The Gray Zone of Faithfulness: Taming Ambiguity in Unfaithfulness Detection
• Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
• Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding
• Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
• GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer
• CPR: Mitigating Large Language Model Hallucinations with Curative Prompt Refinement
• The Curious Case of Factual (Mis)Alignment between LLMs' Short- and Long-Form Answers
• Bolster Hallucination Detection via Prompt-Guided Data Augmentation
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models
• Large Language Models Hallucination: A Comprehensive Survey
• Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrs
• ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
• TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
• Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
• Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality
• Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
• A Novel Differential Feature Learning for Effective Hallucination Detection and Classification
• Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain
• Principled Detection of Hallucinations in Large Language Models via Multiple Testing
• Exploring Causal Effect of Social Bias on Faithfulness Hallucinations in Large Language Models
• SCALEFeedback: A Large-Scale Dataset of Synthetic Computer Science Assignments for LLM-generated Educational Feedback Research
• Dean of LLM Tutors: Exploring Comprehensive and Automated Evaluation of LLM-generated Educational Feedback via LLM Feedback Evaluators
• Towards Reliable Generative AI-Driven Scaffolding: Reducing Hallucinations and Enhancing Quality in Self-Regulated Learning Support
• The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs
• Enhancing Hallucination Detection via Future Context

Related

• Distinguishing Ignorance from Error in LLM Hallucinations
• Fine-grained Hallucination Detection and Editing for Language Models
• LLMs Will Always Hallucinate, and We Need to Live With This
• On Large Language Models' Hallucination with Regard to Known Facts
• Large Language Models Hallucination: A Comprehensive Survey