SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts

2025/07/17 by Brinner, Marc, Zarriess, Sina · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2507.13105

Abstract

We introduce SemCSE, an unsupervised method for learning semantic embeddings of scientific texts. Building on recent advances in contrastive learning for text embeddings, our approach leverages LLM-generated summaries of scientific abstracts to train a model that positions semantically related summaries closer together in the embedding space. This resulting objective ensures that the model captures the true semantic content of a text, in contrast to traditional citation-based approaches that do not necessarily reflect semantic similarity. To validate this, we propose a novel benchmark designed to assess a model's ability to understand and encode the semantic content of scientific texts, demonstrating that our method enforces a stronger semantic separation within the embedding space. Additionally, we evaluate SemCSE on the comprehensive SciRepEval benchmark for scientific text embeddings, where it achieves state-of-the-art performance among models of its size, thus highlighting the benefits of a semantically focused training approach.

Citations

• Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them
• Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance
• Little Giants: Synthesizing High-Quality Embedding Data at Scale
• GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings
• jina-embeddings-v3: Multilingual Embeddings With Task LoRA
• Preserving Multilingual Quality While Tuning Query Encoder on English Only
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents
• SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
• Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
• Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity
• Document Embedding for Scientific Articles: Efficacy of Word Embeddings vs TFIDF
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach
• CLEAR: Contrastive Learning for Sentence Representation
• Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• SPECTER: Document-level Representation Learning using Citation-informed Transformers
• Dense Passage Retrieval for Open-Domain Question Answering
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Representation Degeneration Problem in Training Natural Language Generation Models
• Content-Based Citation Recommendation
• Deep metric learning using Triplet network

Cited by

• SemCSE-Multi: Multifaceted and Decodable Embeddings for Aspect-Specific and Interpretable Scientific Domain Mapping

Related

• SemCSE-Multi: Multifaceted and Decodable Embeddings for Aspect-Specific and Interpretable Scientific Domain Mapping
• MCSE: Multimodal Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Interpretable contrastive word mover's embedding