A Tutorial on Deep Latent Variable Models of Natural Language

2018/12/17 by Yoon Kim, Sam Wiseman, Kim, Yoon +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1812.06834

openalex publication_date 2018/12/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

There has been much recent, exciting work on combining the complementary strengths of latent variable models and deep learning. Latent variable modeling makes it easy to explicitly specify model constraints through conditional independence properties, while deep learning makes it possible to parameterize these conditional likelihoods with powerful function approximators. While these "deep latent variable" models provide a rich, flexible framework for modeling many real-world phenomena, difficulties exist: deep parameterizations of conditional likelihoods usually make posterior inference intractable, and latent variable objectives often complicate backpropagation by introducing points of non-differentiability. This tutorial explores these issues in depth through the lens of variational inference.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Annealed importance sampling
• Recurrent Neural Network Regularization
• Importance Weighted Autoencoders
• Improving neural networks by preventing co-adaptation of feature detectors
• Auto-Encoding Variational Bayes
• Long Short-Term Memory
• Finding Structure in Time
• A Fast Learning Algorithm for Deep Belief Nets
• Adaptive Mixtures of Local Experts
• Variational inference for Monte Carlo objectives
• Sequence-to-Sequence Learning as Beam-Search Optimization
• f-GAN: Training Generative Neural Samplers using Variational Divergence\n Minimization
• Improving Variational Inference with Inverse Autoregressive Flow
• Improved Techniques for Training GANs
• Composing graphical models with neural networks for structured representations and fast inference
• Unsupervised Neural Hidden Markov Models
• Learning in Implicit Generative Models
• Categorical Reparameterization with Gumbel-Softmax
• Variational Graph Auto-Encoders
• Variational Lossy Autoencoder
• Grammar Variational Autoencoder
• Masked Autoregressive Flow for Density Estimation
• Discovering Discrete Latent Topics with Neural Variational Inference
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• InfoVAE: Information Maximizing Variational Autoencoders
• Generating Sentences by Editing Prototypes
• Long Text Generation via Adversarial Training with Leaked Information
• Learnable Explicit Density for Continuous Latent Space and Variational Inference
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Advances in Variational Inference
• Inference Suboptimality in Variational Autoencoders
• Disentangling by Factorising
• Syntax-Directed Variational Autoencoder for Structured Data
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Auto-Encoding Total Correlation Explanation
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Theory and Experiments on Vector Quantized Autoencoders
• Neural Ordinary Differential Equations
• Glow: Generative Flow with Invertible 1x1 Convolutions
• Avoiding Latent Variable Collapse With Generative Skip Models
• Spherical Latent Spaces for Stable Variational Autoencoders
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Pyro: Deep Universal Probabilistic Programming
• On parameter estimation with the Wasserstein distance
• Deconvolutional Latent-Variable Model for Text Sequence Matching
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Adversarial Autoencoders
• Variational Inference: A Review for Statisticians

Related

• Learning with Latent Structures in Natural Language Processing: A Survey
• Improving latent variable descriptiveness with AutoGen
• Latent Variable Modelling Using Variational Autoencoders: A survey
• A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
• Implicit Deep Latent Variable Models for Text Generation