Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

2023/11/28 by Zhiyuan Zhao, Zhao, Zhiyuan, Bin Wang +9 · 29 citations
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Sentiment Analysis and Opinion Mining #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2311.16839

openalex publication_date 2023/11/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multimodal large language models have made significant advancements in recent years, yet they still suffer from a common issue known as the "hallucination problem", in which the models generate textual descriptions that inaccurately depict or entirely fabricate content from associated images. This paper introduces a novel solution, Hallucination-Aware Direct Preference Optimization (HA-DPO), which reframes the hallucination problem as a preference selection task. The model is trained to favor the non-hallucinating response when presented with two responses of the same image (one accurate and one hallucinatory). Furthermore, this paper proposes an efficient pipeline for constructing positive~(non-hallucinatory) and negative~(hallucinatory) sample pairs, ensuring a high-quality, style-consistent dataset for robust preference learning. When applied to three mainstream multimodal models, HA-DPO significantly reduced hallucination issues and amplified the models' generalization capabilities. Notably, the MiniGPT-4 model, when enhanced with HA-DPO, demonstrated a substantial improvement: POPE accuracy rose from 51.13% to 86.13% (an absolute improvement of 35%), and the MME score surged from 932.00 to 1326.46 (a relative improvement of 42.32%). The codes, models, and datasets are made accessible at https://opendatalab.github.io/HA-DPO.

Cited by

• Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection
• Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking
• Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Models
• Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
• OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs
• Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation
• CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
• Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding
• Token-Level Inference-Time Alignment for Vision-Language Models
• Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding
• Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation
• FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models
• Self-Improvement in Multimodal Large Language Models: A Survey
• Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
• Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection
• LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions
• Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
• Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations
• OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
• Controlling Multimodal LLMs via Reward-guided Decoding
• Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
• Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling
• ChartCap: Mitigating Hallucination of Dense Chart Captioning
• MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processing
• See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs
• Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering
• A Survey of Multimodal Hallucination Evaluation and Detection

Related

• Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
• V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization
• TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
• Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
• Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization