Drowning in Documents: Consequences of Scaling Reranker Inference
2024/11/18 by Mathew Jacob, Jacob, Mathew, Erik Lindgren +9 · 7 voices
#cs.IR #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2411.11767
Abstract
Rerankers, typically cross-encoders, are computationally intensive but are frequently used because they are widely assumed to outperform cheaper initial IR systems. We challenge this assumption by measuring reranker performance for full retrieval, not just re-scoring first-stage retrieval. To provide a more robust evaluation, we prioritize strong first-stage retrieval using modern dense embeddings and test rerankers on a variety of carefully chosen, challenging tasks, including internally curated datasets to avoid contamination, and out-of-domain ones. Our empirical results reveal a surprising trend: the best existing rerankers provide initial improvements when scoring progressively more documents, but their effectiveness gradually declines and can even degrade quality beyond a certain limit. We hope that our findings will spur future research to improve reranking.
Discussions
- Information retrieval systems usually operate as a model "cascade" -- fast vector search over billions of documents followed by a more expressive LLM "re-ranking" the resulting top-K. But beware 👻 ! [bsky, 55 points, 3 comments]
- How many documents to retrieve if you plan to use a reranker? This paper shows that while reranking a limited number of documents (K< 100) can often enhance recall, increasing K beyond this threshold [bsky, 10 points, 2 comments]
- Read the paper for more experiments, analysis, and discussion. ArXiv: arxiv.org/abs/2411.11767 HF: huggingface.co/papers/2411.... [bsky, 4 points, 1 comments]
- Drowning in Documents: Consequences of Scaling Reranker Inference [hn, 2 points, 0 comments]
- Side note: It would have been nice to see precision reported in this study so as to best understand the quality of reranking. arxiv.org/abs/2411.11767 [bsky, 0 points, 0 comments]
- Drowning in Documents: Consequences of Scaling Reranker Inference Analyzes challenges of scaling rerankers for large datasets, showing limitations and proposing robust listwise reranking alternatives [bsky, 0 points, 1 comments]
- I'm a big fan of research that backs up my IR prejudices ;P (scarred by a lot of LTR experiments that all failed. I do wonder if in that k=10 space, you might do just as well in a RAG pipeline by re [bsky, 0 points, 0 comments]
Related