MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers

2025/06/18 by Kalra, Jushaan Singh, Zhao, Xinran, Kim, To Eun +3 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Information Retrieval and Search Behavior #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.15862

openalex publication_date 2025/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Retrieval-augmented Generation (RAG) is powerful, but its effectiveness hinges on which retrievers we use and how. Different retrievers offer distinct, often complementary signals: BM25 captures lexical matches; dense retrievers, semantic similarity. Yet in practice, we typically fix a single retriever based on heuristics, which fails to generalize across diverse information needs. Can we dynamically select and integrate multiple retrievers for each individual query, without the need for manual selection? In our work, we validate this intuition with quantitative analysis and introduce mixture of retrievers: a zero-shot, weighted combination of heterogeneous retrievers. Extensive experiments show that such mixtures are effective and efficient: Despite totaling just 0.8B parameters, this mixture outperforms every individual retriever and even larger 7B models by +10.8% and +3.9% on average, respectively. Further analysis also shows that this mixture framework can help incorporate specialized non-oracle human information sources as retrievers to achieve good collaboration, with a 58.9% relative performance improvement over simulated humans alone.

Citations

• LTRR: Learning To Rank Retrievers for LLMs
• Search-o1: Agentic Search-Enhanced Large Reasoning Models
• MixGR: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularity
• Query Routing for Homogeneous Tools: An Instantiation in the RAG Scenario
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• Generative Representational Instruction Tuning
• Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations
• Fine-Tuning LLaMA for Multi-Stage Text Retrieval
• Selecting which Dense Retriever to use for Zero-Shot Search
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Unsupervised Dense Information Retrieval with Contrastive Learning
• Large Dual Encoders Are Generalizable Retrievers
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Decontextualization: Making Sentences Stand-Alone
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Array programming with NumPy
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Fact or Fiction: Verifying Scientific Claims
• MPNet: Masked and Permuted Pre-training for Language Understanding
• SPECTER: Document-level Representation Learning using Citation-informed Transformers
• Dense Passage Retrieval for Open-Domain Question Answering
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Crowdsourcing Multiple Choice Science Questions
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Matplotlib: A 2D Graphics Environment
• Adaptive Mixtures of Local Experts
• UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities
• RouterRetriever: Routing over a Mixture of Expert Embedding Models
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Evaluation of Agents under Simulated AI Marketplace Dynamics
• LTRR: Learning To Rank Retrievers for LLMs

Related

• UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities
• RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
• RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation
• Multi-task retriever fine-tuning for domain-specific and efficient RAG
• PAIRS: Parametric-Verified Adaptive Information Retrieval and Selection for Efficient RAG