Text Embeddings by Weakly-Supervised Contrastive Pre-training

2022/12/07 by Wang, Liang, Yang, Nan, Huang, Xiaolong +5 · 218 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR)

paper · doi:10.48550/arxiv.2212.03533

Abstract

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.

Cited by

• VL-RouterBench: A Benchmark for Vision-Language Model Routing
• Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework
• Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents
• Do Current Retrievers Cover All the Evidence? A Controlled Study of Conjunctive Cross-Page Retrieval
• EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff
• Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders
• JKO-RAG: Distributional Retrieval as Wasserstein Free-Energy Gradient Flow
• Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study
• How important is Recall for Measuring Retrieval Quality?
• Making Large Language Models Efficient Dense Retrievers
• LIR³AG: A Lightweight Rerank Reasoning Strategy Framework for Retrieval-Augmented Generation
• AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
• DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI
• Coarse-to-Fine Open-Set Graph Node Classification with Large Language Models
• FoodLogAthl-218: Constructing a Real-World Food Image Dataset Using Dietary Management Applications
• Citation importance-aware document representation learning for large-scale science mapping
• MMhops-R1: Multimodal Multi-hop Reasoning
• Semantic Grounding Index: Geometric Bounds on Context Engagement in RAG Systems
• Learning to Retrieve with Weakened Labels: Robust Training under Label Noise
• CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning
• Knowledge Diversion for Efficient Morphology Control and Policy Transfer
• Luxical: High-Speed Lexical-Dense Text Embeddings
• Tokenizing Buildings: A Transformer for Layout Synthesis
• ClusterFusion: Hybrid Clustering with Embedding Guidance and LLM Adaptation
• CARL: Criticality-Aware Agentic Reinforcement Learning
• On GRPO Collapse in Search-R1: The Lazy Likelihood-Displacement Death Spiral
• GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplace
• Agentic Policy Optimization via Instruction-Policy Co-Evolution
• Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation
• From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures
• HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documents
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• Comparative Analysis of LoRA-Adapted Embedding Models for Clinical Cardiology Text Representation
• Scalable Parameter-Light Spectral Method for Clustering Short Text Embeddings with a Cohesion-Based Evaluation Metric
• Attention Grounded Enhancement for Visual Document Retrieval
• GRAPHTEXTACK: A Realistic Black-Box Node Injection Attack on LLM-Enhanced GNNs
• CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic
• Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB
• Retrieval as a Decision: Training-Free Adaptive Gating for Efficient RAG
• Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning
• VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semantics
• BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives
• From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
• Structured RAG for Answering Aggregative Questions
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
• Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models
• A Representation Sharpening Framework for Zero Shot Dense Retrieval
• Query Generation Pipeline with Enhanced Answerability Assessment for Financial Information Retrieval
• MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning
• SemBench: A Benchmark for Semantic Query Processing Engines
• LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026
• Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval
• InfoFlow: Reinforcing Search Agent Via Reward Density Optimization
• ReaKase-8B: Legal Case Retrieval via Knowledge and Reasoning Representations with LLMs
• ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents
• Generalized Pseudo-Relevance Feedback
• GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learning
• Galvanic Vestibular Stimulation in Latent Space
• Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
• Multi-Agent Transactive Memory
• Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
• Is Dimensionality a Barrier for Retrieval Models?
• BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• Optimizing Retrieval for RAG via Reinforced Contrastive Learning
• AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages
• Minimizing Human Intervention in Online Classification
• E2Rank: Your Text Embedding can Also be an Effective and Efficient Listwise Reranker
• FAIR-RAG: Faithful Adaptive Iterative Refinement for Retrieval-Augmented Generation
• GigaEmbeddings: Efficient Russian Language Embedding Model
• PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding
• The "Right" Discourse on Migration: Analysing Migration-Related Tweets in Right and Far-Right Political Movements
• Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
• UniMedVL: Unifying Medical Multimodal Understanding And Generation Through Observation-Knowledge-Analysis
• SQuAI: Scientific Question-Answering with Multi-Agent Retrieval-Augmented Generation
• Attention Enhanced Entity Recommendation for Intelligent Monitoring in Cloud Systems
• GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learning
• Think Straight, Stop Smart: Structured Reasoning for Efficient Multi-Hop RAG
• Search Self-play: Pushing the Frontier of Agent Capability without Supervision
• Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
• MENTOR: A Reinforcement Learning Framework for Enabling Tool Use in Small Models via Teacher-Optimized Rewards
• Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction
• Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning
• Rethinking On-policy Optimization for Query Augmentation
• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
• Cost-Aware Retrieval-Augmentation Reasoning Models with Adaptive Retrieval Depth
• An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs
• DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management
• Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report
• Free-Grained Hierarchical Recognition
• Stop-RAG: Value-Based Retrieval Control for Iterative RAG
• Towards Agentic Self-Learning LLMs in Search Environment
• Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation
• Embedding-Based Context-Aware Reranker
• Simple Projection Variants Improve ColBERT Performance
• Reducing Belief Deviation in Reinforcement Learning for Active Reasoning
• REGENT: Relevance-Guided Attention for Entity-Aware Multi-Vector Neural Re-Ranking
• Who are you, ChatGPT? Personality and Demographic Style in LLM-Generated Content
• Domain-Specific Data Generation Framework for RAG Adaptation
• ZeroGR: A Generalizable and Scalable Framework for Zero-Shot Generative Retrieval
• RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generation
• Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learning
• Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
• LinearRAG: Linear Graph Retrieval Augmented Generation on Large-scale Corpora
• HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks
• One Sentence, Two Embeddings: Contrastive Learning of Explicit and Implicit Semantic Representations
• PairSem: LLM-Guided Pairwise Semantic Matching for Scientific Document Retrieval
• ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval
• HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation
• A²Search: Ambiguity-Aware Question Answering with Reinforcement Learning
• Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER
• LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding
• Search-R3: Unifying Reasoning and Embedding in Large Language Models
• Study on LLMs for Promptagator-Style Dense Retriever Training
• TWIST: Training-free and Label-free Short Text Clustering through Iterative Vector Updating with LLMs
• PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs
• Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
• M3Retrieve: Benchmarking Multimodal Retrieval for Medicine
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents
• DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision
• Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
• Compressed Concatenation of Small Embedding Models
• Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewards
• RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback
• A Simple but Effective Elaborative Query Reformulation Approach for Natural Language Recommendation
• ModernVBERT: Towards Smaller Visual Document Retrievers
• LLM Routing with Dueling Feedback
• MASH: Modeling Abstention via Selective Help-Seeking
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards
• Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs
• Optimizing What Matters: AUC-Driven Learning for Robust Neural Retrieval
• CustomIR: Unsupervised Fine-Tuning of Dense Embeddings for Known Document Corpora
• LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science
• Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks
• InfoAgent: Advancing Autonomous Information-Seeking Agents
• PRIVMARK: Private Large Language Models Watermarking with MPC
• Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement
• Predicting Oscar-Nominated Screenplays with Sentence Embeddings
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
• Retrieval-augmented GUI Agents with Generative Guidelines
• Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality
• Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning
• mini-vec2vec: Scaling Universal Geometry Alignment with Linear Transformations
• A Benchmark for Localizing Code and Non-Code Issues in Software Projects
• Tree Search for LLM Agent Reinforcement Learning
• EmbeddingGemma: Powerful and Lightweight Text Representations
• SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs
• Harness-G: A Graph-Structured Harness for Search Agents
• CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
• Group-Reflective Self-Distillation for Agentic Reinforcement Learning
• Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems
• ThreatForest: Multi-Agent Attack Tree Generation with Pluggable TTP Framework Mapping
• Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks
• On The Reproducibility Limitations of RAG Systems
• AIRwaves at CheckThat! 2025: Retrieving Scientific Sources for Implicit Claims on Social Media with Dual Encoders and Neural Re-Ranking
• ReGeS: Reciprocal Retrieval-Generation Synergy for Conversational Recommender Systems
• AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation
• Influence Guided Context Selection for Effective Retrieval-Augmented Generation
• KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning
• Evaluating the Effectiveness and Scalability of LLM-Based Data Augmentation for Retrieval
• SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models
• Patent Language Model Pretraining with ModernBERT
• MARIC: Multi-Agent Reasoning for Image Classification
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs
• Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generation
• Omne-R1: Learning to Reason with Memory for Multi-hop Question Answering
• MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
• PowerChain: A Verifiable Agentic AI System for Automating Distribution Grid Analyses
• Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation
• Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• Boosting Data Utilization for Multilingual Dense Retrieval
• LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
• ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation
• TextlessRAG: End-to-End Visual Document RAG by Speech Without Text
• Understanding the Influence of Synthetic Data for Text Embedders
• Training LLMs to be Better Text Embedders through Bidirectional Reconstruction
• Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
• Grocery to General Merchandise: A Cross-Pollination Recommender using LLMs and Real-Time Cart Context
• IDEAlign: Comparing Large Language Models to Human Experts in Open-ended Interpretive Annotations
• VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
• Re3: Learning to Balance Relevance & Recency for Temporal Information Retrieval
• Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering
• EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes
• Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval
• QZhou-Embedding Technical Report
• Evaluating Recabilities of Foundation Models: A Multi-Domain, Multi-Dataset Benchmark
• Specializing General-purpose LLM Embeddings for Implicit Hate Speech Detection across Datasets
• From Post To Personality: Harnessing LLMs for MBTI Prediction in Social Media
• AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning
• Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities
• AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark
• Granite Embedding R2 Models
• On the Interplay between Musical Preferences and Personality through the Lens of Language
• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• Revisiting RAG Ensemble: A Theoretical and Mechanistic Analysis of Multi-RAG System Collaboration
• MHSNet:An MoE-based Hierarchical Semantic Representation Network for Accurate Duplicate Resume Detection with Large Language Model
• Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Information
• CASPER: Concept-integrated Sparse Representation for Scientific Retrieval
• MCP-Guard: A Defense Framework for Model Context Protocol Integrity in Large Language Model Applications
• ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning
• CLAP: Coreference-Linked Augmentation for Passage Retrieval
• BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation
• GraphProp: Training the Graph Foundation Models using Graph Properties
• An Entity Linking Agent for Question Answering
• Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings
• ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings
• From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model
• MAO-ARAG: Multi-Agent Orchestration for Adaptive Retrieval-Augmented Generation
• Causal2Vec: Improving Decoder-only LLMs as Versatile Embedding Models
• Fine-Grained Privacy Extraction from Retrieval-Augmented Generation Systems via Knowledge Asymmetry Exploitation
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• Uncertainty-driven Embedding Convolution
• FMimic: Foundation Models are Fine-grained Action Learners from Human Videos
• Let It Go? Not Quite: Addressing Item Cold Start in Sequential Recommendations with Content-Based Initialization
• Towards Domain Specification of Embedding Models in Medicine
• TDR: Task-Decoupled Retrieval with Fine-Grained LLM Feedback for In-Context Learning

Related

• Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings
• Improving Text Embeddings with Large Language Models
• Text and Code Embeddings by Contrastive Pre-Training
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• Training Sparse Mixture Of Experts Text Embedding Models