Sparse Overcomplete Word Vector Representations

2015/06/05 by Manaal Faruqui, Yulia Tsvetkov, Faruqui, Manaal +8 · 14 citations
Computer Science · #Advanced Text Analysis Techniques #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1506.02004

Proceedings of ACL 2015

arxiv created 2015/06/05 · arxiv updated 2015/06/08

Abstract

Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We propose methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.

Citations

• The Measurement of Observer Agreement for Categorical Data

Cited by

• SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models
• Understanding Neural Networks through Representation Erasure
• Analysis of Variational Sparse Autoencoders
• Visualizing and Understanding Neural Models in NLP
• CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
• Exploring Task Performance with Interpretable Models via Sparse Auto-Encoders
• Sparse Lifting of Dense Vectors: Unifying Word and Sentence Representations
• FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies
• Dense SAE Latents Are Features, Not Bugs
• Transferring Linear Features Across Language Models With Model Stitching
• Quantifying Structure in CLIP Embeddings: A Statistical Framework for Concept Interpretation
• Learning and Evaluating Sparse Interpretable Sentence Embeddings
• BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models
• Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings
• Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations

Related

• Non-distributional Word Vector Representations
• Sparse Lifting of Dense Vectors: Unifying Word and Sentence Representations
• Word Equations: Inherently Interpretable Sparse Word Embeddingsthrough Sparse Coding
• Advances in Pre-Training Distributed Word Representations
• Efficient Estimation of Word Representations in Vector Space