Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models

2024/05/08 by Luke Merrick, Merrick, Luke, Danmei Xu +5 · 1 voice · 30 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.IR

paper · pdf · doi:10.48550/arxiv.2405.05374

openalex publication_date 2024/05/08 · arxiv published 2024/05/08 · arxiv updated 2024/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This report describes the training dataset creation and recipe behind the family of arctic-embed text embedding models (a set of five models ranging from 22 to 334 million parameters with weights open-sourced under an Apache-2 license). At the time of their release, each model achieved state-of-the-art retrieval accuracy for models of their size on the MTEB Retrieval leaderboard, with the largest model, arctic-embed-l outperforming closed source embedding models such as Cohere's embed-v3 and Open AI's text-embed-3-large. In addition to the details of our training recipe, we have provided several informative ablation studies, which we believe are the cause of our model performance.

Cited by

• Auto-SPT: Automating Semantic Preserving Transformations for Code
• From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures
• Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering
• GigaEmbeddings: Efficient Russian Language Embedding Model
• CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale
• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management
• Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report
• TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation
• Compressed Concatenation of Small Embedding Models
• Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs
• LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations
• MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
• How to Evaluate Medical AI
• Boosting Data Utilization for Multilingual Dense Retrieval
• Chronological Passage Assembling in RAG framework for Temporal Question Answering
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injection
• IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering
• Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI
• Exploiting Leaderboards for Large-Scale Distribution of Malicious Models
• FlexOlmo: Open Language Models for Flexible Data Use
• MedGen: Unlocking Medical Video Generation by Scaling Granularly-annotated Medical Videos
• SweRank: Software Issue Localization with Code Ranking
• Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data
• S-DAT: A Multilingual, GenAI-Driven Framework for Automated Divergent Thinking Assessment
• Evergreen: Efficient Claim Verification for Semantic Aggregates
• Safety Pretraining: Toward the Next Generation of Safe AI
• Generative AI Enhanced Financial Risk Management Information Retrieval

Discussions

• Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models [hn, 1 points, 0 comments]

Related

• Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
• Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
• EmbeddingGemma: Powerful and Lightweight Text Representations
• Improving Text Embeddings with Large Language Models
• F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data