Word Mover's Embedding: From Word2Vec to Document Embedding

2018/10/30 by Lingfei Wu, Ian En-Hsu Yen, Wu, Lingfei +13 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Text and Document Classification Technologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1811.01713

openalex publication_date 2018/10/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While the celebrated Word2Vec technique yields semantically rich representations for individual words, there has been relatively less success in extending to generate unsupervised sentences or documents embeddings. Recent work has demonstrated that a distance measure between documents called Word Mover's Distance (WMD) that aligns semantically similar words, yields unprecedented KNN classification accuracy. However, WMD is expensive to compute, and it is hard to extend its use beyond a KNN classifier. In this paper, we propose the Word Mover's Embedding (WME), a novel approach to building an unsupervised document (sentence) embedding from pre-trained word embeddings. In our experiments on 9 benchmark text classification datasets and 22 textual similarity tasks, the proposed technique consistently matches or outperforms state-of-the-art techniques, with significantly higher accuracy on problems of short length.

Citations

• Term-weighting approaches in automatic text retrieval
• Exploiting Similarities among Languages for Machine Translation
• Convolutional Neural Networks for Sentence Classification

Cited by

• SecureReviewer: Enhancing Large Language Models for Secure Code Review through Secure-aware Fine-tuning
• Speeding up Word Mover's Distance and its variants via properties of distances between embeddings
• Reinforcement Learning Based Text Style Transfer without Parallel Training Corpus
• Exploiting Rich Syntactic Information for Semantic Parsing with Graph-to-Sequence Model

Related

• Speeding up Word Mover's Distance and its variants via properties of distances between embeddings
• Utility of General and Specific Word Embeddings for Classifying Translational Stages of Research
• Moving Other Way: Exploring Word Mover Distance Extensions
• Text classification with word embedding regularization and soft similarity measure
• Deconstructing word embedding algorithms