Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for\n Pairwise Sentence Scoring Tasks

2020/10/16 by Nandan Thakur, Thakur, Nandan, Nils Reimers +5 · 13 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2010.08240

Abstract

There are two approaches for pairwise sentence scoring: Cross-encoders, which\nperform full-attention over the input pair, and Bi-encoders, which map each\ninput independently to a dense vector space. While cross-encoders often achieve\nhigher performance, they are too slow for many practical use cases.\nBi-encoders, on the other hand, require substantial training data and\nfine-tuning over the target task to achieve competitive performance. We present\na simple yet efficient data augmentation strategy called Augmented SBERT, where\nwe use the cross-encoder to label a larger set of input pairs to augment the\ntraining data for the bi-encoder. We show that, in this process, selecting the\nsentence pairs is non-trivial and crucial for the success of the method. We\nevaluate our approach on multiple tasks (in-domain) as well as on a domain\nadaptation task. Augmented SBERT achieves an improvement of up to 6 points for\nin-domain and of up to 37 points for domain adaptation tasks compared to the\noriginal bi-encoder performance.\n

Cited by

• Unified Semantic Modeling Framework for Large-Scale Job Understanding at LinkedIn
• Distance Is All You Need: Radial Dispersion for Uncertainty Estimation in Large Language Models
• WET -- Weighted Ensemble Transformer for Identifying Psychiatric Stressors Related to Suicide on X (formerly Twitter)
• Technique to Baseline QE Artefact Generation Aligned to Quality Metrics
• PolicyBot - Reliable Question Answering over Policy Documents
• Probing Knowledge Holes in Unlearned LLMs
• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• Mnemosyne: An Unsupervised, Human-Inspired Long-Term Memory Architecture for Edge-Based LLMs
• KurdSTS: The Kurdish Semantic Textual Similarity
• Uncovering the Bigger Picture: Comprehensive Event Understanding Via Diverse News Retrieval
• Statistical Comparative Analysis of Semantic Similarities and Model Transferability Across Datasets for Short Answer Grading
• Do Biased Models Have Biased Thoughts?
• LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations

Related

• Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
• Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
• Can Cross Encoders Produce Useful Sentence Embeddings?
• Sentence Pair Scoring: Towards Unified Framework for Text Comprehension
• Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling