Sparse, Dense, and Attentional Representations for Text Retrieval

2020/05/01 by Yi Luan, Luan, Yi, Jacob Eisenstein +5 · 44 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #Machine Learning and Algorithms #cs.CL

paper · pdf · doi:10.48550/arxiv.2005.00181

To appear in TACL 2020. The arXiv version is a pre-MIT Press publication version

arxiv created 2021/02/16 · arxiv updated 2021/02/18

Abstract

Dual encoders perform retrieval by encoding documents and queries into dense lowdimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.

Citations

• Type-Token Mathematics: A Textbook of Mathematical Linguistics.
• Database-friendly random projections: Johnson-Lindenstrauss with binary coins
• A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL

Cited by

• SMART: LLM-Augmented Hybrid Retrieval for Dynamic Product Ads
• CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• CroPS: Improving Dense Retrieval with Cross-Perspective Positive Samples in Short-Video Search
• Optimizing Dense Retrieval Model Training with Hard Negatives
• RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Distilling Knowledge from Reader to Retriever for Question Answering
• A Memory Efficient Baseline for Open Domain Question Answering
• CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale
• Understanding Hard Negatives in Noise Contrastive Estimation
• Dense Text Retrieval Based on Pretrained Language Models: A Survey
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking
• Condenser: a Pre-training Architecture for Dense Retrieval
• PairSem: LLM-Guided Pairwise Semantic Matching for Scientific Document Retrieval
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization
• From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text
• UniDex: Rethinking Search Inverted Indexing with Unified Semantic Modeling
• Investigating Multi-layer Representations for Dense Passage Retrieval
• LexSemBridge: Fine-Grained Dense Representation Enhancement through Token-Aware Embedding Augmentation
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• OPERA: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• RRRA: Resampling and Reranking through a Retriever Adapter
• Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Search
• An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking
• Matching-oriented Product Quantization For Ad-hoc Retrieval
• SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
• Semantic Certainty Assessment in Vector Retrieval Systems: A Novel Framework for Embedding Quality Evaluation
• Generation-Augmented Retrieval for Open-domain Question Answering
• From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems
• Ultra-Fast, Low-Storage, Highly Effective Coarse-grained Selection in Retrieval-based Chatbot by Using Deep Semantic Hashing
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• CoRet: Improved Retriever for Code Editing
• Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models
• Scientific Paper Retrieval with LLM-Guided Semantic-Based Ranking
• Hybrid Encoder: Towards Efficient and Precise Native AdsRecommendation via Hybrid Transformer Encoding Networks
• Mitigating the Position Bias of Transformer Models in Passage Re-Ranking
• Select-And-Extract: A Lightweight Plugin for Retrieval-Augmented Generation
• Evidence-Unit Fairness and the Limits of Query-Adaptive Sparse-Dense Fusion in Financial Document Retrieval
• COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
• ReasonIR: Training Retrievers for Reasoning Tasks
• Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation
• Towards Lossless Token Pruning in Late-Interaction Retrieval Models

Related

• What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary
• Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval
• Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval
• On Single and Multiple Representations in Dense Passage Retrieval
• Investigating Multi-layer Representations for Dense Passage Retrieval