HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models

2025/06/21 by Le Yu, Kaishen Wang, Yu, Le +8 · 2 citations
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2506.17587

openalex publication_date 2025/06/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Though Large Vision-Language Models (LVLMs) have achieved remarkable performance across various tasks, they are still prone to hallucinations-generating outputs that are textually plausible but visually ungrounded. While prior approaches generally address this issue through data-centric fine-tuning or innovative decoding strategies, these methods often require substantial resources or task-specific configurations. In this work, we introduce an architecture-level solution, HalluRNN, which enhances model stability through recurrent cross-layer reasoning. Specifically, we propose a novel Dual-Gated Depth Propagation Unit (DG-DPU) module, which is shared across layers and recurrently refines hidden states. This allows for the adaptive propagation of information throughout the model, enforces consistency across layers, and mitigates hallucinations caused by representational drift. By fine-tuning only the DG-DPU module, HalluRNN achieves strong and robust performance across multiple benchmarks.

Citations

• ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation
• MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
• Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Models
• MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLMs Will Always Hallucinate, and We Need to Live With This
• Hallucination Detection in LLMs: Fast and Memory-Efficient Fine-Tuned Models
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
• Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
• The Llama 3 Herd of Models
• Strengthening Layer Interaction via Dynamic Layer Attention
• Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models
• Mitigating Large Language Model Hallucination with Faithful Finetuning
• Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
• CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models
• Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
• Calibrated Self-Rewarding Vision Language Models
• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
• Unveiling the Generalization Power of Fine-Tuned Large Language Models
• IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
• Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
• Hallucination is Inevitable: An Innate Limitation of Large Language Models
• Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites
• RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
• LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
• CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• 🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
• Detecting and Preventing Hallucinations in Large Vision Language Models
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• Evaluating Object Hallucination in Large Vision-Language Models
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Visual Instruction Tuning
• Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT
• Cross-Layer Retrospective Retrieving via Layer Attention
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Two-stage LLM Fine-tuning with Less Specialization and More Generalization
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
• Survey of Hallucination in Natural Language Generation
• BA-Net: Bridge Attention for Deep Convolutional Neural Networks
• Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
• Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning
• Advancing RNN Transducer Technology for Speech Recognition
• Language Models are Few-Shot Learners
• Towards Causal VQA: Revealing and Reducing Spurious Correlations by\n Invariant and Covariant Semantic Editing
• DIANet: Dense-and-Implicit Attention Network
• Object Hallucination in Image Captioning
• Comparative Study of CNN and RNN for Natural Language Processing
• Densely Connected Convolutional Networks
• Analyzing the Behavior of Visual Question Answering Models
• Deep Residual Learning for Image Recognition
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Microsoft COCO: Common Objects in Context
• Long Short-Term Memory
• Debiasing Multimodal Large Language Models via Penalization of Language Priors
• Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Cited by

• MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processing
• Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

Related

• Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
• Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning
• VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
• Reducing Hallucinations in Vision-Language Models via Latent Space Steering