Exploring Causal Effect of Social Bias on Faithfulness Hallucinations in Large Language Models

2025/08/11 by Zhenliang Zhang, Zhang, Zhenliang, Junzhe Zhang +7
Computer Science · Psychology · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Mental Health via Writing #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.07753

openalex publication_date 2025/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have achieved remarkable success in various tasks, yet they remain vulnerable to faithfulness hallucinations, where the output does not align with the input. In this study, we investigate whether social bias contributes to these hallucinations, a causal relationship that has not been explored. A key challenge is controlling confounders within the context, which complicates the isolation of causality between bias states and hallucinations. To address this, we utilize the Structural Causal Model (SCM) to establish and validate the causality and design bias interventions to control confounders. In addition, we develop the Bias Intervention Dataset (BID), which includes various social biases, enabling precise measurement of causal effects. Experiments on mainstream LLMs reveal that biases are significant causes of faithfulness hallucinations, and the effect of each bias state differs in direction. We further analyze the scope of these causal effects across various models, specifically focusing on unfairness hallucinations, which are primarily targeted by social bias, revealing the subtle yet significant causal effect of bias on hallucination generation.

Citations

• ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs
• Gemma 2: Improving Open Language Models at a Practical Size
• Look Within, Why LLMs Hallucinate: A Causal Perspective
• Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models
• Breaking Bias, Building Bridges: Evaluation and Mitigation of Social Biases in LLMs via Contact Hypothesis
• Mitigating Large Language Model Hallucination with Faithful Finetuning
• Fairness in Large Language Models: A Taxonomic Survey
• Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generation
• "Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters
• Gender bias and stereotypes in Large Language Models
• Lost in the Middle: How Language Models Use Long Contexts
• Sources of Hallucination by Large Language Models on Inference Tasks
• How Language Model Hallucinations Can Snowball
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
• Large Language Models Can Be Easily Distracted by Irrelevant Context
• Survey of Hallucination in Natural Language Generation
• BBQ: A Hand-Built Bias Benchmark for Question Answering
• Gender Bias in Machine Translation
• Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
• Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages

Related

• Do Large Language Models Show Biases in Causal Learning?
• A review of faithfulness metrics for hallucination assessment in Large Language Models
• Large Language Models Hallucination: A Comprehensive Survey
• Mind vs. Mouth: On Measuring Re-judge Inconsistency of Social Bias in Large Language Models
• When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs