WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach

2021/04/05 by Junjie Huang, Duyu Tang, Huang, Junjie +13 · 6 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2104.01767

Abstract

Producing the embedding of a sentence in an unsupervised way is valuable to natural language matching and retrieval problems in practice. In this work, we conduct a thorough examination of pretrained model based unsupervised sentence embeddings. We study on four pretrained models and conduct massive experiments on seven datasets regarding sentence semantics. We have there main findings. First, averaging all tokens is better than only using [CLS] vector. Second, combining both top andbottom layers is better than only using top layers. Lastly, an easy whitening-based vector normalization strategy with less than 10 lines of code consistently boosts the performance.

Citations

• LLM-Supported Manufacturing Mapping Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Cited by

• Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?
• Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB
• Demystifying Network Foundation Models
• Digital Voices of Survival: From Social Media Disclosures to Support Provisions for Domestic Violence Victims
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey

Related

• Whitening Sentence Representations for Better Semantics and Faster Retrieval
• A Simple and Plug-and-play Method for Unsupervised Sentence Representation Enhancement
• Generating Datasets with Pretrained Language Models
• UNSEE: Unsupervised Non-contrastive Sentence Embeddings
• Whitening Not Recommended for Classification Tasks in LLMs