Multimodal Word Distributions

2017/04/27 by Ben Athiwaratkun, Andrew Gordon Wilson, Athiwaratkun, Ben +1 · 2 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1704.08424

This paper also appears at ACL 2017

openalex publication_date 2017/04/27 · arxiv created 2019/09/09 · arxiv updated 2019/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective. We show that the resulting approach captures uniquely expressive semantic information, and outperforms alternatives, such as word2vec skip-grams, and Gaussian embeddings, on benchmark datasets such as word similarity and entailment.

Citations

• Efficient Estimation of Word Representations in Vector Space
• The Proof and Measurement of Association between Two Things
• Gaussian Mixture Embeddings for Multiple Word Prototypes
• Stochastic Variational Deep Kernel Learning
• Bayesian Recurrent Neural Networks
• Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Cited by

• Mixtures Closest to a Given Measure: A Semidefinite Programming Approach
• Efficient Graph-based Word Sense Induction by Distributional Inclusion Vector Embeddings

Related

• Word Representations via Gaussian Embedding
• Learning Multi-Sense Word Distributions using Approximate Kullback-Leibler Divergence
• Gaussian Word Embedding with a Wasserstein Distance Loss
• Embedding Words as Distributions with a Bayesian Skip-gram Model
• Gaussian Mixture Embeddings for Multiple Word Prototypes