All-but-the-Top: Simple and Effective Postprocessing for Word Representations

2017/02/05 by Mu, Jiaqi, Bhat, Suma, Viswanath, Pramod · 13 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1702.01417

Abstract

Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a \em very simple, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a few top dominating directions from the word vectors -- that renders off-the-shelf representations \em even stronger. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and text classification) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

Cited by

• The Hitchhiker's Guide to Monoculture
• Distribution-Aware Exploration for Adaptive HNSW Search
• One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces
• Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB
• TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domain
• A short survey on almost orthogonal vectors in a few specific large dimensions
• Linear Dimensionality Reduction for Word Embeddings in Tabular Data Classification
• IDEAlign: Comparing Large Language Models to Human Experts in Open-ended Interpretive Annotations
• A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension
• Metis: Training LLMs with FP4 Quantization
• Factor Augmented Supervised Learning with Text Embeddings
• Combining Discrete Wavelet and Cosine Transforms for Efficient Sentence Embedding
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal

Related

• Effect of Post-processing on Contextualized Word Representations
• An Empirical Study on Post-processing Methods for Word Embeddings
• Sparse Overcomplete Word Vector Representations
• Extrofitting: Enriching Word Representation and its Vector Space with Semantic Lexicons
• Advances in Pre-Training Distributed Word Representations