PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers

2024/02/13 by Weizhe Lin, Lin, Weizhe, Jingbiao Mei +5 · 8 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2402.08327

openalex publication_date 2024/02/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pre-trained version of the recently developed Fine-grained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in general-purpose multi-modal retrievers.

Cited by

• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
• Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering
• FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detection
• Taming a Retrieval Framework to Read Images in Humanlike Manner for Augmenting Generation of MLLMs
• Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
• Recurrence Meets Transformers for Universal Multimodal Retrieval
• Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey

Related

• Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering
• Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering
• OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
• SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
• mKG-RAG: Multimodal Knowledge Graph-Enhanced RAG for Visual Question Answering