Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models

2023/05/25 by Doostmohammadi, Ehsan, Norlund, Tobias, Kuhlmann, Marco +1 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2305.16243

Abstract

Augmenting language models with a retrieval mechanism has been shown to significantly improve their performance while keeping the number of parameters low. Retrieval-augmented models commonly rely on a semantic retrieval mechanism based on the similarity between dense representations of the query chunk and potential neighbors. In this paper, we study the state-of-the-art Retro model and observe that its performance gain is better explained by surface-level similarities, such as token overlap. Inspired by this, we replace the semantic retrieval in Retro with a surface-level method based on BM25, obtaining a significant reduction in perplexity. As full BM25 retrieval can be computationally costly for large datasets, we also apply it in a re-ranking scenario, gaining part of the perplexity reduction with minimal computational overhead.

Cited by

• Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency

Related

• On the Generalization Ability of Retrieval-Enhanced Transformers
• SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction
• Retrieval-Augmented Generation for Large Language Models: A Survey
• RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing
• Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models