Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models

2025/06/26 by Jiahe Chen, Chen, Jiahe, Jiaying He +14 · 1 citation
Biochemistry, Genetics and Molecular Biology · Neuroscience · #Brain Tumor Detection and Classification #Cell Image Analysis Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Functional Brain Connectivity Studies

paper · pdf · doi:10.48550/arxiv.2506.21509

openalex publication_date 2025/06/26 · openalex created_date 2025/10/15 · openalex updated_date 2026/08/02

Abstract

Large Vision-Language Models (LVLMs) face a tug-of-war between powerful linguistic priors and visual evidence, often leading to semantic drift: a progressive detachment from the input image that can abruptly emerge at specific decoding steps. Through a token-level diagnosis, we show that hallucination is frequently triggered not by the absence of grounded candidates, but by a failure of selection -- the model chooses a linguistically convenient yet visually unfaithful token even when better grounded alternatives exist. Motivated by this insight, we propose Dynamic Logits Calibration (DLC), a training-free decoding framework that introduces a lightweight visual referee to intervene exactly when drift happens. At each step, DLC performs a dual-aspect grounding check on top-k candidates: (1) it assesses the intrinsic visual relevance of a candidate token and (2) its contextual visual coherence. These signals are evaluated against an adaptive historical baseline to compute a relative visual advantage, which is then used to dynamically calibrate logits and favor grounded tokens. Extensive experiments on CHAIR, POPE, SHR, GPT-4o evaluation, and MME demonstrate that DLC consistently reduces hallucinations across multiple LVLMs while preserving response quality. Further analyses validate robustness to different vision backbones and demonstrate a favorable trade-off between output quality and computational cost as the candidate pool size varies. Code will be released on https://github.com/JiaheChen2002/DLC.

Citations

• FG-CLIP: Fine-Grained Visual and Textual Alignment
• PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
• MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
• Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification
• Detecting and Evaluating Medical Hallucinations in Large Vision Language Models
• Hallucination of Multimodal Large Language Models: A Survey
• Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
• Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
• HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
• Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
• Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
• Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
• RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
• Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
• mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
• Improved Baselines with Visual Instruction Tuning
• Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
• Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
• Qwen Technical Report
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• LISA: Reasoning Segmentation via Large Language Model
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Evaluating Object Hallucination in Large Vision-Language Models
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Visual Instruction Tuning
• Sigmoid Loss for Language Image Pre-Training
• LLaMA: Open and Efficient Foundation Language Models
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Flamingo: a Visual Language Model for Few-Shot Learning
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Learning Transferable Visual Models From Natural Language Supervision
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Object Hallucination in Image Captioning
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Microsoft COCO: Common Objects in Context
• VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models

Related

• Revisit What You See: Disclose Language Prior in Vision Tokens for LVLM Decoding
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
• VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering
• Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs