Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering

2025/08/31 by Changin Choi, Wonseok Lee, Choi, Changin +5 · 1 voice
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.AI #cs.CV

paper · pdf · doi:10.48550/arxiv.2509.00798

arxiv published 2025/08/31 · arxiv updated 2026/04/22

Abstract

Knowledge-intensive visual question answering (VQA) requires external knowledge beyond image content, demanding precise visual grounding and coherent integration of visual and textual information. Although multimodal retrieval-augmented generation has achieved notable advances by incorporating external knowledge bases, existing approaches largely adopt single-pass frameworks that often fail to acquire sufficient knowledge and lack mechanisms to revise misdirected reasoning. We propose PMSR (Progressive Multimodal Search and Reasoning), a framework that progressively constructs a structured reasoning trajectory to enhance both knowledge acquisition and synthesis. PMSR uses dual-scope queries conditioned on both the latest record and the trajectory to retrieve diverse knowledge from heterogeneous knowledge bases. The retrieved evidence is then synthesized into compact records via compositional reasoning. This design facilitates controlled iterative refinement, which supports more stable reasoning trajectories with reduced error propagation. Extensive experiments across six diverse benchmarks (Encyclopedic-VQA, InfoSeek, MMSearch, LiveVQA, FVQA, and OK-VQA) demonstrate that PMSR consistently improves both retrieval recall and end-to-end answer accuracy.

Citations

• Understanding the Effects of Distractors on Reasoning Vision-Language Models
• Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
• DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
• MMSearch-R1: Incentivizing LMMs to Search
• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
• s3: You Don't Need That Much Data to Train a Search Agent via RL
• OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
• MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework
• Seeking and Updating with Live Visual Knowledge
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
• Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval
• LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers
• Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant
• mR²AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
• MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines
• Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions
• MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training
• Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach
• Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
• RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback
• RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation
• PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
• UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context
• Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
• End-to-end Knowledge Retrieval with Multi-modal Queries
• Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
• Large Language Models Can Be Easily Distracted by Irrelevant Context
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering
• Dense Passage Retrieval for Open-Domain Question Answering
• OK-VQA: A Visual Question Answering Benchmark Requiring External\n Knowledge
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Discussions

• Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering - By arXiv:2509.00798v3 https://arxiv.org/abs/2509.00798 MI-RAG boosts visual Q&A by iteratively fetching and refining extern [bsky, 0 points, 0 comments]

Related

• mKG-RAG: Multimodal Knowledge Graph-Enhanced RAG for Visual Question Answering
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
• Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
• OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
• Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries