CLEAR: Contrastive Learning for Sentence Representation

2020/12/31 by Zhuofeng Wu, Sinong Wang, Wu, Zhuofeng +9 · 230 citations
Computer Science · Psychology · #Artificial intelligence #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Law #Linguistics #Natural Language Processing Techniques #Natural language processing #Philosophy #Political science #Psychology #Representation (politics) #Sentence #Sentiment Analysis and Opinion Mining #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2012.15466

published in arXiv (Cornell University) (Cornell University) · 10 pages, 2 figures

arxiv created 2020/12/31 · openalex publication_date 2020/12/31 · arxiv updated 2021/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

Citations

• Adam: A Method for Stochastic Optimization
• Pointer Sentinel Mixture Models
• An efficient framework for learning sentence representations
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Contrastive Multiview Coding
• A large annotated corpus for learning natural language inference
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Certified Robustness to Adversarial Word Substitutions
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning

Cited by

• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• Learning Representations Through Contrastive Neural Model Checking
• Semi-Supervised Synthetic Data Generation with Fine-Grained Relevance Control for Short Video Search Relevance Modeling
• Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Latent Expression Generation for Referring Image Segmentation and Grounding
• SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation
• False Negative Distillation and Contrastive Learning for Personalized Outfit Recommendation
• SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• RadBERT-CL: Factually-Aware Contrastive Learning For Radiology Report Classification
• Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering
• Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learning
• CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
• Unsupervised Document Embedding via Contrastive Augmentation
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning
• CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models
• AimTS: Augmented Series and Image Contrastive Learning for Time Series Classification
• Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generation

Related

• Differentiable Data Augmentation for Contrastive Sentence Representation Learning
• Contrastive Learning of Sentence Embeddings from Scratch
• DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning