CLEAR: Contrastive Learning for Sentence Representation

2020/12/31 by Zhuofeng Wu, Sinong Wang, Wu, Zhuofeng +9 · 7 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Sentiment Analysis and Opinion Mining #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2012.15466

openalex publication_date 2020/12/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

Citations

• Adam: A Method for Stochastic Optimization
• Pointer Sentinel Mixture Models
• An efficient framework for learning sentence representations
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Contrastive Multiview Coding
• A large annotated corpus for learning natural language inference
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Certified Robustness to Adversarial Word Substitutions
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning

Cited by

• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• Learning Representations Through Contrastive Neural Model Checking
• Semi-Supervised Synthetic Data Generation with Fine-Grained Relevance Control for Short Video Search Relevance Modeling
• Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Latent Expression Generation for Referring Image Segmentation and Grounding
• SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation

Related

• Differentiable Data Augmentation for Contrastive Sentence Representation Learning
• Contrastive Learning of Sentence Embeddings from Scratch
• DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning