The Influence of Down-Sampling Strategies on SVD Word Embedding\n Stability

2018/08/21 by Johannes Hellrich, Hellrich, Johannes, Bernd Kampe +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1808.06810

openalex publication_date 2018/08/21 · openalex created_date 2022/08/04 · openalex updated_date 2026/07/28

Abstract

The stability of word embedding algorithms, i.e., the consistency of the word\nrepresentations they reveal when trained repeatedly on the same data set, has\nrecently raised concerns. We here compare word embedding algorithms on three\ncorpora of different sizes, and evaluate both their stability and accuracy. We\nfind strong evidence that down-sampling strategies (used as part of their\ntraining procedures) are particularly influential for the stability of\nSVDPPMI-type embeddings. This finding seems to explain diverging reports on\ntheir stability and lead us to a simple modification which provides superior\nstability as well as accuracy on par with skip-gram embeddings.\n

Citations

• Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
• Distributional Structure

Related

• Are Word Embedding Methods Stable and Should We Care About It?
• Word Embeddings: Stability and Semantic Change
• On the Convergent Properties of Word Embedding Methods
• Understanding the Downstream Instability of Word Embeddings
• On the Downstream Performance of Compressed Word Embeddings