Domain-matched Pre-training Tasks for Dense Retrieval

2021/07/28 by Oğuz, Barlas, Lakhotia, Kushal, Gupta, Anchit +8 · 3 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR)

paper · doi:10.48550/arxiv.2107.13602

Abstract

Pre-training on larger datasets with ever increasing model size is now a proven recipe for increased performance across almost all NLP tasks. A notable exception is information retrieval, where additional pre-training has so far failed to produce convincing results. We show that, with the right pre-training setup, this barrier can be overcome. We demonstrate this by pre-training large bi-encoder models on 1) a recently released set of 65 million synthetically generated questions, and 2) 200 million post-comment pairs from a preexisting dataset of Reddit conversations made available by pushshift.io. We evaluate on a set of information retrieval and dialogue retrieval benchmarks, showing substantial improvements over supervised baselines.

Cited by

• CroPS: Improving Dense Retrieval with Cross-Perspective Positive Samples in Short-Video Search
• RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
• Investigating Multi-layer Representations for Dense Passage Retrieval

Related

• REALM: Retrieval-Augmented Language Model Pre-Training
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition
• Large Language Models Struggle to Learn Long-Tail Knowledge
• Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval