PromptBERT: Improving BERT Sentence Embeddings with Prompts

2022/01/12 by Ting Jiang, Jiang, Ting, Jiao, Jian +15 · 5 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Sentiment Analysis and Opinion Mining

paper · pdf · doi:10.48550/arxiv.2201.04337

Abstract

We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.

Cited by

• An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
• Hierarchical Section Matching Prediction (HSMP) BERT for Fine-Grained Extraction of Structured Data from Hebrew Free-Text Radiology Reports in Crohn's Disease
• CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similarity
• Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey

Related

• Self-Guided Contrastive Learning for BERT Sentence Representations
• CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thought
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• 2-Tier SimCSE: Elevating BERT for Robust Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings