VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation

2024/11/18 by Ruiyang Zhang, Hu Zhang, Zhang, Ruiyang +3 · 18 citations
Computer Science · #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2411.11919

openalex publication_date 2024/11/18 · openalex created_date 2024/11/21 · openalex updated_date 2026/07/28

Abstract

Given the higher information load processed by large vision-language models (LVLMs) compared to single-modal LLMs, detecting LVLM hallucinations requires more human and time expense, and thus rise a wider safety concerns. In this paper, we introduce VL-Uncertainty, the first uncertainty-based framework for detecting hallucinations in LVLMs. Different from most existing methods that require ground-truth or pseudo annotations, VL-Uncertainty utilizes uncertainty as an intrinsic metric. We measure uncertainty by analyzing the prediction variance across semantically equivalent but perturbed prompts, including visual and textual data. When LVLMs are highly confident, they provide consistent responses to semantically equivalent queries. However, when uncertain, the responses of the target LVLM become more random. Considering semantically similar answers with different wordings, we cluster LVLM responses based on their semantic content and then calculate the cluster distribution entropy as the uncertainty measure to detect hallucination. Our extensive experiments on 10 LVLMs across four benchmarks, covering both free-form and multi-choice tasks, show that VL-Uncertainty significantly outperforms strong baseline methods in hallucination detection.

Cited by

• Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking
• LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems
• AlignBench: Benchmarking Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs
• When to Trust the Answer: Question-Aligned Semantic Nearest Neighbor Entropy for Safer Surgical VQA
• When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs
• HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Models
• Semantic Reformulation Entropy for Robust Hallucination Detection in QA Tasks
• GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
• Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• A Survey of Multimodal Hallucination Evaluation and Detection
• Analysis of Image-and-Text Uncertainty Propagation in Multimodal Large Language Models with Cardiac MR-Based Applications
• Legal knowledge infusion for large language models: A survey
• Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models
• Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
• MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM

Related

• Reference-free Hallucination Detection for Large Vision-Language Models
• Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directions
• A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs
• Revisiting Hallucination Detection with Effective Rank-based Uncertainty
• Leveraging NTPs for Efficient Hallucination Detection in VLMs