Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings

2025/09/16 by Shi-Yu Li, Li, Shiyu, Yang Tang +6 · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Law #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.12892

openalex publication_date 2025/09/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have recently demonstrated excellent performance in text embedding tasks. Previous work usually use LoRA to fine-tune existing LLMs, which are limited by the data and training gap between LLMs and embedding models. In this work, we introduce Conan-embedding-v2, a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder. First, we add news data and multilingual pairs for LLM pretraining to bridge the data gap. Based on this, we propose a cross-lingual retrieval dataset that enables the LLM to better integrate embeddings across different languages. Second, whereas LLMs use a causal mask with token-level loss, embedding models use a bidirectional mask with sentence-level loss. This training gap makes full fine-tuning less effective than LoRA. We introduce a soft-masking mechanism to gradually transition between these two types of masks, enabling the model to learn more comprehensive representations. Based on this, we propose a dynamic hard negative mining method that exposes the model to more difficult negative examples throughout the training process. Being intuitive and effective, with only approximately 1.4B parameters, Conan-embedding-v2 achieves SOTA performance on both the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025).

Citations

• Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languages
• Making Text Embedders Few-Shot Learners
• Conan-embedding: General Text Embedding with More and Better Negative Samples
• mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
• NV-Retriever: Improving text embedding models with effective hard-negative mining
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
• LoRA Learns Less and Forgets Less
• Understanding Cross-Lingual Alignment -- A Survey
• Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• T2Ranking: A large-scale Chinese Benchmark for Passage Ranking
• Unsupervised Dense Information Retrieval with Contrastive Learning
• Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
• LoRA: Low-Rank Adaptation of Large Language Models
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews
• TWEAC: Transformer with Extendable QA Agent Classifiers
• OCNLI: Original Chinese Natural Language Inference
• The Multilingual Amazon Reviews Corpus
• MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• Dense Passage Retrieval for Open-Domain Question Answering
• Efficient Intent Detection with Dual Sentence Encoders
• Scaling Laws for Neural Language Models
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• PubMedQA: A Dataset for Biomedical Research Question Answering
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• FEVER: a large-scale dataset for Fact Extraction and VERification
• DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Distributed Representations of Words and Phrases and their Compositionality
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards

Related

• LLMs are Also Effective Embedding Models: An In-depth Overview
• Conan-embedding: General Text Embedding with More and Better Negative Samples
• KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
• Improving Text Embeddings with Large Language Models
• Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks