2015/02/13 by David Belanger, Sham Kakade, Sham M. Kakade +2 · 2 citations
Computer Science · Mathematics · Physics and Astronomy · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Opinion Dynamics and Social Influence #Topic Modeling #cs.CL #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1502.04081
Accepted at International Conference of Machine Learning 2015
openalex publication_date 2015/02/13 · arxiv created 2015/05/31 · arxiv updated 2015/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Low dimensional representations of words allow accurate NLP models to be trained on limited annotated data. While most representations ignore words' local context, a natural way to induce context-dependent representations is to perform inference in a probabilistic latent-variable sequence model. Given the recent success of continuous vector space word representations, we provide such an inference procedure for continuous states, where words' representations are given by the posterior mean of a linear dynamical system. Here, efficient inference can be performed using Kalman filtering. Our learning algorithm is extremely scalable, operating on simple cooccurrence counts for both parameter initialization using the method of moments and subsequent iterations of EM. In our experiments, we employ our inferred word embeddings as features in standard tagging tasks, obtaining significant accuracy improvements. Finally, the Kalman filter updates can be seen as a linear recurrent neural network. We demonstrate that using the parameters of our model to initialize a non-linear recurrent neural network language model reduces its training time by a day and yields lower perplexity.