2025/01/26 by Xiaohan Yu, Zhihan Yang, Yu, Xiaohan +3 · 1 voice · 1 citation
Computer Science · #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Multiagent Systems (cs.MA) #Speech and dialogue systems #cs.IR #cs.MA
paper · pdf · doi:10.48550/arxiv.2501.15470
openalex publication_date 2025/01/26 · arxiv published 2025/01/26 · openalex created_date 2025/10/10 · arxiv updated 2025/10/31 · openalex updated_date 2026/08/03
Multimodal Retrieval Augmented Generation (MRAG) systems have shown promise in enhancing the generation capabilities of multimodal large language models (MLLMs). However, existing MRAG frameworks primarily adhere to rigid, single-step retrieval strategies that fail to address real-world challenges of information acquisition and query reformulation. In this work, we introduce the task of Multimodal Retrieval Augmented Generation Planning (MRAG Planning) that aims at effective information seeking and integration while minimizing computational overhead. Specifically, we propose CogPlanner, an agentic plug-and-play framework inspired by human cognitive processes, which iteratively determines query reformulation and retrieval strategies to generate accurate and contextually relevant responses. CogPlanner supports parallel and sequential modeling paradigms. Furthermore, we introduce CogBench, a new benchmark designed to rigorously evaluate the MRAG Planning task and facilitate lightweight CogPlanner integration with resource-efficient MLLMs, such as Qwen2-VL-7B-Cog. Experimental results demonstrate that CogPlanner significantly outperforms existing MRAG baselines, offering improvements in both accuracy and efficiency with minimal additional computational costs.