Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models

2024/05/08 by Merrick, Luke, Xu, Danmei, Nuti, Gaurav +1 · 19 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR)

paper · doi:10.48550/arxiv.2405.05374

Abstract

This report describes the training dataset creation and recipe behind the family of arctic-embed text embedding models (a set of five models ranging from 22 to 334 million parameters with weights open-sourced under an Apache-2 license). At the time of their release, each model achieved state-of-the-art retrieval accuracy for models of their size on the MTEB Retrieval leaderboard, with the largest model, arctic-embed-l outperforming closed source embedding models such as Cohere's embed-v3 and Open AI's text-embed-3-large. In addition to the details of our training recipe, we have provided several informative ablation studies, which we believe are the cause of our model performance.

Cited by

• Auto-SPT: Automating Semantic Preserving Transformations for Code
• From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures
• Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering
• GigaEmbeddings: Efficient Russian Language Embedding Model
• CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale
• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management
• Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report
• TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation
• Compressed Concatenation of Small Embedding Models
• The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs
• LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations
• MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
• How to Evaluate Medical AI
• Boosting Data Utilization for Multilingual Dense Retrieval
• Chronological Passage Assembling in RAG framework for Temporal Question Answering
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering

Related

• Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
• Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
• EmbeddingGemma: Powerful and Lightweight Text Representations
• Improving Text Embeddings with Large Language Models
• F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data