Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

2026/08/03 by Jingchen Sun, Shaobo Han, Ruiyi Zhang +5
Computer Science · #cs.LG

paper · pdf

arxiv created 2026/08/03 · arxiv updated 2026/08/05

Abstract

Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs. However, existing contrastive training methods typically treat all unmatched query-document pairs as equally informative negatives, which is problematic because many unmatched documents may still be semantically relevant or partially useful. We propose Bayesian Data Reweighting, a probabilistic framework that models query-document importance as latent variables and adaptively infers posterior weights to downweight likely false negatives. With closed-form posterior updates under conjugate priors and stochastic EM optimization, our method consistently improves retrieval accuracy across three retrievers and seven knowledge-based VQA benchmarks.

Citations

• Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval
• Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
• PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
• UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
• Improved Baselines with Visual Instruction Tuning
• Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
• Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
• REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
• Retrieval Augmented Visual Question Answering with Outside Knowledge
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
• Incremental False Negative Detection for Contrastive Learning
• Learning Transferable Visual Models From Natural Language Supervision
• Boosting Contrastive Self-Supervised Learning with False Negative Cancellation
• Contrastive Learning with Hard Negative Samples
• Hard Negative Mixing for Contrastive Learning
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• OK-VQA: A Visual Question Answering Benchmark Requiring External\n Knowledge
• Hardness-Aware Deep Metric Learning
• From Recognition to Cognition: Visual Commonsense Reasoning
• Representation Learning with Contrastive Predictive Coding
• Learning to Reweight Examples for Robust Deep Learning
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Sampling Matters in Deep Embedding Learning
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Convergence of a stochastic approximation version of the EM algorithm