Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores

2024/08/19 by Junwei Lu, Lu, Jun, David C. S. Li +4 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2408.11868

openalex publication_date 2024/08/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper presents an approach to improve text embedding models through contrastive fine-tuning on small datasets augmented with expert scores. It focuses on enhancing semantic textual similarity tasks and addressing text retrieval problems. The proposed method uses soft labels derived from expert-augmented scores to fine-tune embedding models, preserving their versatility and ensuring retrieval capability is improved. The paper evaluates the method using a Q&A dataset from an online shopping website and eight expert models. Results show improved performance over a benchmark model across multiple metrics on various retrieval tasks from the massive text embedding benchmark (MTEB). The method is cost-effective and practical for real-world applications, especially when labeled data is scarce.

Cited by

• Scalable Unit Harmonization in Medical Informatics via Bayesian-Optimized Retrieval and Transformer-Based Re-ranking

Related

• Efficient fine-tuning methodology of text embedding models for information retrieval: contrastive learning penalty (clp)
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework
• Training Sparse Mixture Of Experts Text Embedding Models
• Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark