MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering

2025/11/15 by Seokwon Song, Song, Seokwon, Minsu Park +3
Computer Science · #Advanced Graph Neural Networks #Attribution #Benchmark (surveying) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Key (lock) #Multimodal Machine Learning Applications #Multimodal interaction #Multimodality #Question answering #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2511.12142

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/11/15 · openalex created_date 2025/11/19 · openalex updated_date 2026/08/05

Abstract

Source attribution aims to enhance the reliability of AI-generated answers by including references for each statement, helping users validate the provided answers. However, existing work has primarily focused on text-only scenario and largely overlooked the role of multimodality. We introduce MAVIS, the first benchmark designed to evaluate multimodal source attribution systems that understand user intent behind visual questions, retrieve multimodal evidence, and generate long-form answers with citations. Our dataset comprises 157K visual QA instances, where each answer is annotated with fact-level citations referring to multimodal documents. We develop fine-grained automatic metrics along three dimensions of informativeness, groundedness, and fluency, and demonstrate their strong correlation with human judgments. Our key findings are threefold: (1) LVLMs with multimodal RAG generate more informative and fluent answers than unimodal RAG, but they exhibit weaker groundedness for image documents than for text documents, a gap amplified in multimodal settings. (2) Given the same multimodal documents, there is a trade-off between informativeness and groundedness across different prompting methods. (3) Our proposed method highlights mitigating contextual bias in interpreting image documents as a crucial direction for future research.

Citations

• Qwen3 Technical Report
• Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning
• Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
• MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation
• Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling
• Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Long-Form Answers to Visual Questions from Blind and Low Vision People
• LLaVA-OneVision: Easy Visual Task Transfer
• The Llama 3 Herd of Models
• RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
• CoTAR: Chain-of-Thought Attribution Reasoning with Multi-level Granularity
• Improving Attributed Text Generation of Large Language Models via Preference Learning
• Attribute First, then Generate: Locally-attributable Grounded Text Generation
• Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification
• Training Language Models to Generate Text with Citations via Fine-grained Rewards
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin
• Enabling Large Language Models to Generate Text with Citations
• Evaluating Verifiability in Generative Search Engines
• Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
• Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
• Measuring Attribution in Natural Language Generation Models
• Measuring Attribution in Natural Language Generation Models
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• HowSumm: A Multi-Document Summarization Dataset Derived from WikiHow Articles
• MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
• AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• ELI5: Long Form Question Answering

Related

• Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering
• Multimodal Rationales for Explainable Visual Question Answering
• VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering
• Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning
• Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries