Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens

2025/08/04 by Zhenguo Zhang, Zheng, Haohan, Zhang, Zhenguo · 3 citations
Arts and Humanities · Computer Science · Engineering · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Multimodal Machine Learning Applications #Subtitles and Audiovisual Media

paper · pdf · doi:10.48550/arxiv.2508.02419

openalex publication_date 2025/08/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large vision-language models (LVLMs) have demonstrated remarkable multimodal comprehension and reasoning capabilities, but they still suffer from severe object hallucination. Previous studies primarily attribute the flaw to linguistic prior caused by the scale mismatch between visual encoders and large language models (LLMs) in LVLMs. Specifically, as current LVLMs are built upon LLMs, they tend to over-rely on textual prompts and internal knowledge of LLMs, generating descriptions inconsistent with visual cues. However, through an in-depth investigation of the hallucinated mechanisms, we empirically reveal a previously overlooked phenomenon: LVLMs may ignore not only visual information but also textual modality during hallucination, a behavior termed as modality bias, which indicates that LVLMs struggle to simultaneously attend to both visual and textual modalities, leading to fragmented understanding of user-provided instructions. Based on this observation, we propose a simple yet effective training-free method to mitigate object hallucination. Concretely, we intervene and adjust the attention weights of textual and visual tokens, balancing cross-modal compatibility for better alignment with user intentions. Furthermore, we adopt a contrastive decoding strategy to reduce the LVLM's overreliance on its parametric knowledge, synergistically enhancing our attention manipulation. Extensive experiments confirm the widespread presence of modality bias in LVLMs. Notably, our method effectively mitigates hallucination across multiple open-source LVLMs and benchmarks, highlighting its generalizability and efficacy.

Citations

• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
• Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
• Reducing Hallucinations in Vision-Language Models via Latent Space Steering
• MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
• VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment
• A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
• Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps
• HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
• A Survey on Multimodal Large Language Models for Autonomous Driving
• Improved Baselines with Visual Instruction Tuning
• Efficient Streaming Language Models with Attention Sinks
• Vision Transformers Need Registers
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
• MMBench: Is Your Multi-modal Model an All-around Player?
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• LLaMA: Open and Efficient Foundation Language Models
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
• Fine-grained Image Captioning with CLIP Reward
• Deduplicating Training Data Makes Language Models Better
• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• Object Hallucination in Image Captioning
• Attention Is All You Need
• Microsoft COCO: Common Objects in Context

Cited by

• OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
• MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoning
• A Survey of Multimodal Hallucination Evaluation and Detection
• System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

Related

• Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
• Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
• Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs