Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models

2025/06/29 by Iturra-Bocaz, Gabriel, Vo, Danny, Galuscakova, Petra
#FOS: Computer and information sciences #Information Retrieval (cs.IR)

paper · doi:10.48550/arxiv.2506.23191

Abstract

This paper investigates the impact of shallow versus deep relevance judgments on the performance of BERT-based reranking models in neural Information Retrieval. Shallow-judged datasets, characterized by numerous queries each with few relevance judgments, and deep-judged datasets, involving fewer queries with extensive relevance judgments, are compared. The research assesses how these datasets affect the performance of BERT-based reranking models trained on them. The experiments are run on the MS MARCO and LongEval collections. Results indicate that shallow-judged datasets generally enhance generalization and effectiveness of reranking models due to a broader range of available contexts. The disadvantage of the deep-judged datasets might be mitigated by a larger number of negative training examples.

Citations

• Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection
• LongEval-Retrieval: French-English Dynamic Test Collection for Continuous Web Search Evaluation
• Hard Negatives or False Negatives: Correcting Pooling Bias in Training Neural Ranking Models
• Overview of the TREC 2020 deep learning track
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Overview of the TREC 2019 deep learning track
• Document Ranking with a Pretrained Sequence-to-Sequence Model
• Multi-Stage Document Ranking with BERT
• Passage Re-ranking with BERT
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
• The Evolution of Reranking Models in Information Retrieval: From Heuristic Methods to Large Language Models
• How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
• ReFIT: Relevance Feedback from a Reranker during Inference
• Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking