When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR

2025/06/02 by Dayoon Ko, Ko, Dayoon, Jin-Young Kim +12 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.01877

openalex publication_date 2025/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Dense retrievers encode texts into embeddings to efficiently retrieve relevant documents from large databases in response to user queries. However, real-world corpora continually evolve, leading to a shift from the original training distribution of the retriever. Without timely updates or retraining, indexing newly emerging documents can degrade retrieval performance for future queries. Thus, identifying when a dense retriever requires an update is critical for maintaining robust retrieval systems. In this paper, we propose a novel task of predicting whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. Addressing this task allows us to proactively manage retriever updates, preventing potential retrieval failures. We introduce GradNormIR, an unsupervised approach that leverages gradient norms to detect OOD corpora effectively. Experiments on the BEIR benchmark demonstrate that GradNormIR enables timely updates of dense retrievers in evolving document collections, significantly enhancing retrieval robustness and efficiency.

Citations

• Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective
• GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?
• Multi-Head RAG: Solving Multi-Aspect Problems with LLMs
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• Selecting which Dense Retriever to use for Zero-Shot Search
• Continual Learning for Generative Retrieval over Dynamic Corpora
• L²R: Lifelong Learning for First-stage Retrieval with Backward-Compatible Representations
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• Lightweight Learner for Shared Knowledge Lifelong Learning
• Unsupervised Layer-wise Score Aggregation for Textual OOD Detection
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• Large Language Models with Controllable Working Memory
• Augmenting Document Representations for Dense Retrieval with\n Interpolation and Perturbation
• Unsupervised Dense Information Retrieval with Contrastive Learning
• Large Dual Encoders Are Generalizable Retrievers
• GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval
• On the Importance of Gradients for Detecting Distributional Shifts in the Wild
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• Optimizing Dense Retrieval Model Training with Hard Negatives
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
• Dense Passage Retrieval for Open-Domain Question Answering
• How Context Affects Language Models' Factual Predictions
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
• Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median
• RouterRetriever: Routing over a Mixture of Expert Embedding Models

Cited by

• Evaluating the Robustness of Dense Retrievers in Interdisciplinary Domains

Related

• When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets
• Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
• Exploring the Practicality of Generative Retrieval on Dynamic Corpora
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• CustomIR: Unsupervised Fine-Tuning of Dense Embeddings for Known Document Corpora