AugCSE: Contrastive Sentence Embedding with Diverse Augmentations

2022/10/20 by Zilu Tang, Muhammed Yusuf Kocyigit, Tang, Zilu +3 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2210.13749

openalex publication_date 2022/10/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Data augmentation techniques have been proven useful in many applications in NLP fields. Most augmentations are task-specific, and cannot be used as a general-purpose tool. In our work, we present AugCSE, a unified framework to utilize diverse sets of data augmentations to achieve a better, general purpose, sentence embedding model. Building upon the latest sentence embedding models, our approach uses a simple antagonistic discriminator that differentiates the augmentation types. With the finetuning objective borrowed from domain adaptation, we show that diverse augmentations, which often lead to conflicting contrastive signals, can be tamed to produce a better and more robust sentence representation. Our methods achieve state-of-the-art results on downstream transfer tasks and perform competitively on semantic textual similarity tasks, using only unsupervised data.

Cited by

• Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?

Related

• Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning
• DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
• Contrastive Learning of Sentence Embeddings from Scratch
• SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learning
• SimCSE: Simple Contrastive Learning of Sentence Embeddings