Condenser: a Pre-training Architecture for Dense Retrieval

2021/04/16 by Luyu Gao, Gao, Luyu, Jamie Callan +1 · 7 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2104.08253

openalex publication_date 2021/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.

Citations

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Sparse, Dense, and Attentional Representations for Text Retrieval
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers

Cited by

• LLMs as Sparse Retrievers:A Framework for First-Stage Product Search
• Investigating Multi-layer Representations for Dense Passage Retrieval
• Effect of Model Merging in Domain-Specific Ad-hoc Retrieval
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• Training LLMs to be Better Text Embedders through Bidirectional Reconstruction
• SEAL: Structure and Element Aware Learning to Improve Long Structured Document Retrieval
• RRRA: Resampling and Reranking through a Retriever Adapter

Related

• Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
• Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder
• Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval
• Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval
• Dense Text Retrieval Based on Pretrained Language Models: A Survey