UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers

2023/03/01 by Saad-Falcon, Jon, Khattab, Omar, Santhanam, Keshav +6 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR)

paper · doi:10.48550/arxiv.2303.00807

Abstract

Many information retrieval tasks require large labeled datasets for fine-tuning. However, such datasets are often unavailable, and their utility for real-world applications can diminish quickly due to domain shifts. To address this challenge, we develop and motivate a method for using large language models (LLMs) to generate large numbers of synthetic queries cheaply. The method begins by generating a small number of synthetic queries using an expensive LLM. After that, a much less expensive one is used to create large numbers of synthetic queries, which are used to fine-tune a family of reranker models. These rerankers are then distilled into a single efficient retriever for use in the target domain. We show that this technique boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods.

Cited by

• IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering
• Evaluating the Robustness of Dense Retrievers in Interdisciplinary Domains

Related

• Enhancing Transformer-Based Rerankers with Synthetic Data and LLM-Based Supervision
• DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation
• InRanker: Distilled Rankers for Zero-shot Information Retrieval
• LLM Optimization Unlocks Real-Time Pairwise Reranking
• How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models