Unsupervised Document Embedding via Contrastive Augmentation

2021/03/26 by Dongsheng Luo, Luo, Dongsheng, Wei Cheng +19
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text and Document Classification Technologies #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2103.14542

13 pages; under review

arxiv created 2021/03/26 · openalex publication_date 2021/03/26 · arxiv updated 2021/03/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a contrasting learning approach with data augmentation techniques to learn document representations in an unsupervised manner. Inspired by recent contrastive self-supervised learning algorithms used for image and NLP pretraining, we hypothesize that high-quality document embedding should be invariant to diverse paraphrases that preserve the semantics of the original document. With different backbones and contrastive learning frameworks, our study reveals the enormous benefits of contrastive augmentation for document representation learning with two additional insights: 1) including data augmentation in a contrastive way can substantially improve the embedding quality in unsupervised document representation learning, and 2) in general, stochastic augmentations generated by simple word-level manipulation work much better than sentence-level and document-level ones. We plug our method into a classifier and compare it with a broad range of baseline methods on six benchmark datasets. Our method can decrease the classification error rate by up to 6.4% over the SOTA approaches on the document classification task, matching or even surpassing fully-supervised methods.

Citations

• Skip-Thought Vectors
• Distributed Representations of Sentences and Documents
• Distributed Representations of Words and Phrases and their Compositionality
• Efficient Vector Representation for Documents through Corruption
• Improved Regularization of Convolutional Neural Networks with Cutout
• Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• A Simple Framework for Contrastive Learning of Visual Representations
• Longformer: The Long-Document Transformer
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Exploring Simple Siamese Representation Learning
• CLEAR: Contrastive Learning for Sentence Representation

Related

• Self-supervised Document Clustering Based on BERT with Data Augment
• Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings
• Contextual Document Embeddings
• Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learning
• Contrastive Document Representation Learning with Graph Attention Networks