Streaming Word Embeddings with the Space-Saving Algorithm

2017/04/24 by Chandler May, May, Chandler, Kevin Duh +5 · 1 citation
Computer Science · #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.6 #I.2.7 #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1704.07463

openalex publication_date 2017/04/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We develop a streaming (one-pass, bounded-memory) word embedding algorithm based on the canonical skip-gram with negative sampling algorithm implemented in word2vec. We compare our streaming algorithm to word2vec empirically by measuring the cosine similarity between word pairs under each algorithm and by applying each algorithm in the downstream task of hashtag prediction on a two-month interval of the Twitter sample stream. We then discuss the results of these experiments, concluding they provide partial validation of our approach as a streaming replacement for word2vec. Finally, we discuss potential failure modes and suggest directions for future work.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Finding repeated elements
• Random sampling with a reservoir
• Incremental Skip-gram Model with Negative Sampling
• Distributional Structure

Cited by

• RiverText: A Python Library for Training and Evaluating Incremental Word Embeddings from Text Data Streams

Related

• Learning Word Embeddings from the Portuguese Twitter Stream: A Study of some Practical Aspects
• Learning Word Embedding with Better Distance Weighting and Window Size Scheduling
• Deconstructing word embedding algorithms
• On SkipGram Word Embedding Models with Negative Sampling: Unified Framework and Impact of Noise Distributions
• Corrected CBOW Performs as well as Skip-gram