MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

2024/10/10 by Wenbo Hu, Jia-Chen Gu, Hu, Wenbo +11 · 20 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Geographic Information Systems Studies #Multimodal Machine Learning Applications #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2410.08182

openalex publication_date 2024/10/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Existing multimodal retrieval benchmarks primarily focus on evaluating whether models can retrieve and utilize external textual knowledge for question answering. However, there are scenarios where retrieving visual information is either more beneficial or easier to access than textual data. In this paper, we introduce a multimodal retrieval-augmented generation benchmark, MRAG-Bench, in which we systematically identify and categorize scenarios where visually augmented knowledge is better than textual knowledge, for instance, more images from varying viewpoints. MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. With MRAG-Bench, we conduct an evaluation of 10 open-source and 4 proprietary large vision-language models (LVLMs). Our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge, confirming that MRAG-Bench is vision-centric. Additionally, we conduct extensive analysis with MRAG-Bench, which offers valuable insights into retrieval-augmented LVLMs. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82% improvement with ground-truth information, in contrast to a 33.16% improvement observed in human participants. These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs' ability to utilize retrieved visual knowledge more effectively.

Cited by

• VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents
• Exploration of Augmentation Strategies in Multi-modal Retrieval-Augmented Generation for the Biomedical Domain: A Case Study Evaluating Question Answering in Glycobiology
• M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG
• HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation
• Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation
• Open Multimodal Retrieval-Augmented Factual Image Generation
• MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval
• CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation
• TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models
• MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation
• CMRAG: Co-modality-based visual document retrieval and question answering
• FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data
• Large Language Models Facilitate Vision Reflection in Image Classification
• CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG
• VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval
• Towards General Continuous Memory for Vision-Language Models
• Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
• UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities
• The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
• Mixture-of-RAG: Integrating Text and Tables with Large Language Models

Related

• mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
• Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
• OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
• Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
• Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook