Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers

2026/07/30 by Fexiang Liu, Shiye Wang, Qiang Qiu +1
Computer Science · #cs.CV

paper · pdf

22 pages, 6 figures; includes supplementary material. Code: https://github.com/SouthWinter/WEP

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Reliable deployment of multimodal large language models (MLLMs) requires deciding whether a confident visual answer should be trusted, reviewed, or routed to a stronger system. Confidence scores capture candidate margins, but not where the estimated signed visual readouts associated with those margins come from or how they are distributed. We study inference-time risk detection for closed visual answers using the same white-box prefill path that produces the answer. Witness Evidence Portfolios (WEP) first estimates, layer by layer, which visual contributions support or contradict the predicted candidate. It summarizes these contributions through two interpretable route families: question-related evidence provenance and signed evidence concentration. Nested grouped validation chooses the more reliable family and a sparse top-k route portfolio, which is fused with candidate confidence. WEP needs no image perturbation, decoding change, backward pass, or external verifier. Across three MLLMs and four binary-answer benchmarks, WEP improves mean error AP by 0.134. All 12 model--dataset gains are positive, and image-cluster bootstrap intervals are strictly positive on 10 pairs. WEP targets white-box closed-answer systems and uses a labeled calibration slice.

Citations

• LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems
• HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
• Taming Object Hallucinations with Verified Atomic Confidence Estimation
• Reallocating Attention Across Layers to Reduce Multimodal Hallucination
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• A Survey of Multimodal Hallucination Evaluation and Detection
• Variational Visual Question Answering for Uncertainty-Aware Selective Prediction
• Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
• INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
• Improved Baselines with Visual Instruction Tuning
• MMBench: Is Your Multi-modal Model an All-around Player?
• Improving Selective Visual Question Answering by Learning from Your Peers
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly
• Transformer Interpretability Beyond Attention Visualization
• Attention is not not Explanation
• Attention is not Explanation
• Object Hallucination in Image Captioning
• On Calibration of Modern Neural Networks
• Selective Classification for Deep Neural Networks
• Axiomatic Attribution for Deep Networks
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Related

• Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
• Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
• MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models
• Peek-a-Boo Reasoning: Contrastive Region Masking in MLLMs
• Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning