Simple and Effective Dimensionality Reduction for Word Embeddings

2017/08/11 by Vikas Raunak, Raunak, Vikas · 15 citations
Computer Science · Mathematics · #Artificial intelligence #Computation and Language (cs.CL) #Computer science #Curse of dimensionality #Dimension (graph theory) #Dimensionality reduction #Embedding #FOS: Computer and information sciences #Mathematics #Natural Language Processing Techniques #Natural language processing #Reduction (mathematics) #Similarity (geometry) #Text and Document Classification Technologies #Theoretical computer science #Topic Modeling #Word (group theory) #Word embedding #Word processing #cs.CL

paper · pdf · doi:10.48550/arxiv.1708.03629

published in arXiv (Cornell University) (Cornell University) · Accepted at NIPS 2017 LLD Workshop

openalex publication_date 2017/08/11 · arxiv created 2017/11/21 · arxiv updated 2017/11/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Word embeddings have become the basic building blocks for several natural language processing and information retrieval tasks. Pre-trained word embeddings are used in several downstream applications as well as for constructing representations for sentences, paragraphs and documents. Recently, there has been an emphasis on further improving the pre-trained word vectors through post-processing algorithms. One such area of improvement is the dimensionality reduction of the word embeddings. Reducing the size of word embeddings through dimensionality reduction can improve their utility in memory constrained devices, benefiting several real-world applications. In this work, we present a novel algorithm that effectively combines PCA based dimensionality reduction with a recently proposed post-processing algorithm, to construct word embeddings of lower dimensions. Empirical evaluations on 12 standard word similarity benchmarks show that our algorithm reduces the embedding dimensionality by 50%, while achieving similar or (more often) better performance than the higher dimension embeddings.

Citations

• A Tutorial on Principal Component Analysis

Cited by

• A Large-Language-Model Framework for Automated Humanitarian Situation Reporting
• Selecting User Histories to Generate LLM Users for Cold-Start Item Recommendation
• Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks
• Combining Discrete Wavelet and Cosine Transforms for Efficient Sentence Embedding
• Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform
• InsurTech innovation using natural language processing
• Including Semantic Information via Word Embeddings for Skeleton-based Action Recognition
• The Future is Sparse: Embedding Compression for Scalable Retrieval in Recommender Systems
• Contextual Embedding-based Clustering to Identify Topics for Healthcare Service Improvement

Related

• Word Embedding Dimension Reduction via Weakly-Supervised Feature Selection
• Single Training Dimension Selection for Word Embedding with PCA
• Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings
• An Empirical Study on Post-processing Methods for Word Embeddings
• Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings