State Space LSTM Models with Particle MCMC Inference

2017/11/30 by Xun Zheng, Zheng, Xun, Manzil Zaheer +9 · 1 citation
Computer Science · Materials Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1711.11179

openalex publication_date 2017/11/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Long Short-Term Memory (LSTM) is one of the most powerful sequence models. Despite the strong performance, however, it lacks the nice interpretability as in state space models. In this paper, we present a way to combine the best of both worlds by introducing State Space LSTM (SSL) models that generalizes the earlier work \citezaheer2017latent of combining topic models with LSTM. However, unlike \citezaheer2017latent, we do not make any factorization assumptions in our inference algorithm. We present an efficient sampler based on sequential Monte Carlo (SMC) method that draws from the joint posterior directly. Experimental results confirms the superiority and stability of this SMC inference algorithm on a variety of domains.

Citations

• Auto-Encoding Variational Bayes
• Maximum likelihood estimates of linear dynamic systems
• Black box variational inference for state space models
• Variational Sequential Monte Carlo
• Auto-Encoding Sequential Monte Carlo
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data

Cited by

• Variational Dynamic Mixtures

Related

• Improving the Performance of the LSTM and HMM Model via Hybridization
• Empirical Evaluation of A New Approach to Simplifying Long Short-term Memory (LSTM)
• Towards Binary-Valued Gates for Robust LSTM Training
• Scalable Monte Carlo inference for state-space models
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs