MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation

2025/02/06 by Qinhan Yu, Zhi‐You Xiao, Yu, Qinhan +9 · 12 citations
Computer Science · #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.04176

openalex publication_date 2025/02/06 · openalex created_date 2025/02/08 · openalex updated_date 2026/07/28

Abstract

Recent advances in Retrieval-Augmented Generation (RAG) have significantly improved response accuracy and relevance by incorporating external knowledge into Large Language Models (LLMs). However, existing RAG methods primarily focus on generating text-only answers, even in Multimodal Retrieval-Augmented Generation (MRAG) scenarios, where multimodal elements are retrieved to assist in generating text answers. To address this, we introduce the Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) task, in which we aim to generate multimodal answers that combine both text and images, fully leveraging the multimodal data within a corpus. Despite growing attention to this challenging task, a notable lack of a comprehensive benchmark persists for effectively evaluating its performance. To bridge this gap, we provide MRAMG-Bench, a meticulously curated, human-annotated benchmark comprising 4,346 documents, 14,190 images, and 4,800 QA pairs, distributed across six distinct datasets and spanning three domains: Web, Academia, and Lifestyle. The datasets incorporate diverse difficulty levels and complex multi-image scenarios, providing a robust foundation for evaluating the MRAMG task. To facilitate rigorous evaluation, MRAMG-Bench incorporates a comprehensive suite of both statistical and LLM-based metrics, enabling a thorough analysis of the performance of generative models in the MRAMG task. Additionally, we propose an efficient and flexible multimodal answer generation framework that can leverage LLMs/MLLMs to generate multimodal responses. Our datasets and complete evaluation results for 11 popular generative models are available at https://github.com/MRAMG-Bench/MRAMG.

Cited by

• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research
• M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG
• Epistemic Substitution: How Grokipedia's AI-Generated Encyclopedia Restructures Authority
• MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering
• Caption Injection for Optimization in Generative Search Engine
• Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation
• Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrieval
• CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation
• VIG-RL: Learning to Search and Insert for Verified Image Grounding
• Reinforcement Learning Foundations for Deep Research Systems: A Survey
• M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generation
• Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?

Related

• A Survey of Multimodal Retrieval-Augmented Generation
• Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
• Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text