Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models

2025/01/02 by Huang, Yanwen, Zhang, Yong, Cheng, Ning +3 · 4 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2501.01059

Abstract

Large language models (LLMs) often exhibit Context Faithfulness Hallucinations, where outputs deviate from retrieved information due to incomplete context integration. Our analysis reveals a strong correlation between token-level uncertainty and hallucinations. We hypothesize that attention mechanisms inherently encode context utilization signals, supported by probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD), a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. Experiments on open-book QA datasets demonstrate DAGCD's effectiveness, yielding significant improvements in faithfulness and robustness while preserving computational efficiency.

Cited by

• COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation
• Probing Latent Knowledge Conflict for Faithful Retrieval-Augmented Generation
• Large Language Models Hallucination: A Comprehensive Survey
• HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models

Related

• COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation
• HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models
• Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models
• Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
• Maintaining Informative Coherence: Migrating Hallucinations in Large Language Models via Absorbing Markov Chains