A Recurrent Latent Variable Model for Sequential Data

2015/06/07 by Junyoung Chung, Jun‐Young Chung, Kyle Kastner +10 · 706 citations
Computer Science · #Algorithm #Artificial intelligence #Artificial neural network #Autoencoder #Computer science #Generative Adversarial Networks and Image Synthesis #Hidden variable theory #Latent variable #Latent variable model #Machine learning #Neural Networks and Applications #Recurrent neural network #Speech recognition #State (computer science) #State variable #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.1506.02216

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2015/06/07 · arxiv created 2016/04/06 · arxiv updated 2016/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we explore the inclusion of latent random variables into the dynamic hidden state of a recurrent neural network (RNN) by combining elements of the variational autoencoder. We argue that through the use of high-level latent random variables, the variational RNN (VRNN)1 can model the kind of variability observed in highly structured sequential data such as natural speech. We empirically evaluate the proposed model against related sequential models on four speech datasets and one handwriting dataset. Our results show the important roles that latent random variables can play in the RNN dynamic hidden state.

Citations

• Auto-Encoding Variational Bayes

Cited by

• Label-free Industrial Fault Detection via Adversarial Inverse Reinforcement Learning: A System for Run-to-Failure Prognostics
• Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classification
• Generative Recursive Reasoning
• Inference-time Stochastic Refinement of GRU-Normalizing Flow for Real-time Video Motion Transfer
• Leveraging unlabelled data for generalizable neural population decoding
• Neural Stochastic Flows: Solver-Free Modelling and Inference for SDE Solutions
• Variational Transformers for Diverse Response Generation
• Variational Sequential Labelers for Semi-Supervised Learning
• baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling
• Learning on predictions: Fusing training and autoregressive inference for long-term spatiotemporal forecasts
• A Survey on Bayesian Deep Learning
• Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows
• Analysis of ODE2VAE with Examples
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Variational Marginal Particle Filters
• Continual Learning in Recurrent Neural Networks
• Latent-Variable Generative Models for Data-Efficient Text Classification
• Bayesian Attention Modules
• Mind the Gap when Conditioning Amortised Inference in Sequential\n Latent-Variable Models
• Contrastive Variational Reinforcement Learning for Complex Observations
• Towards Identifiability of Hierarchical Temporal Causal Representation Learning
• Vector Quantization in the Brain: Grid-like Codes in World Models
• CAMNet: Leveraging Cooperative Awareness Messages for Vehicle Trajectory Prediction
• UF-RNN: Real-Time Adaptive Motion Generation Using Uncertainty-Driven Foresight Prediction
• Memory-Augmented Generative AI for Real-time Wireless Prediction in Dynamic Industrial Environments
• Curiosity-Driven Development of Action and Language in Robots Through Self-Exploration
• Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledge
• Neural Bayesian Filtering
• Dynamical system reconstruction from partial observations using stochastic dynamics
• EgoTraj-Bench: Towards Robust Trajectory Prediction Under Ego-view Noisy Observations
• S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling
• Recurrent Independent Mechanisms
• On Predictive Information in RNNs
• Transferable Time-Series Forecasting under Causal Conditional Shift
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Latent Matters: Learning Deep State-Space Models
• BRAID: Input-Driven Nonlinear Dynamical Modeling of Neural-Behavioral Data
• Hierarchical Variational Imitation Learning of Control Programs
• Online Variational Filtering and Parameter Learning
• Deep Variational Luenberger-type Observer for Stochastic Video Prediction
• Sequential Adversarial Anomaly Detection for One-Class Event Data
• SoundSpaces: Audio-Visual Navigation in 3D Environments
• Sequential Neural Processes
• Semi-supervised Sequential Generative Models
• Graph Generation with Variational Recurrent Neural Network
• A Classifying Variational Autoencoder with Application to Polyphonic Music Generation
• Contrastively Disentangled Sequential Variational Autoencoder
• Goal-Directed Planning by Reinforcement Learning and Active Inference
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Memory and attention in deep learning
• MemoryKT: An Integrative Memory-and-Forgetting Method for Knowledge Tracing
• FairDRL-ST: Disentangled Representation Learning for Fair Spatio-Temporal Mobility Prediction
• Probabilistic Video Generation using Holistic Attribute Control
• L-GTA: Latent Generative Modeling for Time Series Augmentation
• Learning representations for multivariate time series with missing data using Temporal Kernelized Autoencoders
• SchrödingeRNN: Generative Modeling of Raw Audio as a Continuously Observed Quantum State
• DSBERT:Unsupervised Dialogue Structure learning with BERT
• Variational Recurrent Models for Solving Partially Observable Control Tasks
• Structured Object-Aware Physics Prediction for Video Modeling and Planning
• Behavior Priors for Efficient Reinforcement Learning
• Hierarchical Video Generation for Complex Data
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• Dynamic Relational Inference in Multi-Agent Trajectories
• Diversifying Topic-Coherent Response Generation for Natural Multi-turn Conversations
• Guiding Variational Response Generator to Exploit Persona
• 3D Conceptual Design Using Deep Learning
• Action2video: Generating Videos of Human 3D Actions
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Variational Hyper RNN for Sequence Modeling
• Generative Adversarial Network for Handwritten Text
• Variational Graph Recurrent Neural Networks
• InfoDPCCA: Information-Theoretic Dynamic Probabilistic Canonical Correlation Analysis
• Monte Carlo Filtering Objectives: A New Family of Variational Objectives to Learn Generative Model and Neural Adaptive Proposal for Time Series
• Bayesian neural networks and dimensionality reduction
• A Generalizable Physics-Enhanced State Space Model for Long-Term Dynamics Forecasting in Complex Environments
• Dynamic Future Net: Diversified Human Motion Generation
• Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
• Probability Trajectory: One New Movement Description for Trajectory Prediction
• H-VGRAE: A Hierarchical Stochastic Spatial-Temporal Embedding Method for Robust Anomaly Detection in Dynamic Networks
• Towards Foundation Auto-Encoders for Time-Series Anomaly Detection
• Recurrent Attentive Neural Process for Sequential Data
• Variational Dynamic Mixtures
• Deep learning reveals hidden interactions in complex systems
• Vertical-Horizontal Structured Attention for Generating Music with Chords
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Wavelet-based Disentangled Adaptive Normalization for Non-stationary Times Series Forecasting
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Generating Handwriting via Decoupled Style Descriptors
• VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• Temporal Difference Variational Auto-Encoder
• Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
• Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
• Ensemble Kalman Variational Objectives: Nonlinear Latent Trajectory Inference with A Hybrid of Variational Inference and Ensemble Kalman Filter
• Disentangled State Space Representations
• Stochastic Sequential Neural Networks with Structured Inference
• Variational Saccading: Efficient Inference for Large Resolution Images
• Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• SimProcess: High Fidelity Simulation of Noisy ICS Physical Processes
• Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer
• An RNN-based IMM Filter Surrogate
• Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networks
• Recovering Hidden Degrees of Freedom Using Gaussian Processes
• Learning Plannable Representations with Causal InfoGAN
• Deep Stochastic Volatility Model
• Learning Latent Variable Models via Jarzynski-adjusted Langevin Algorithm
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Neural Language Generation: Formulation, Methods, and Evaluation
• Alternators With Noise Models
• Recurrent Flow Networks: A Recurrent Latent Variable Model for Density Modelling of Urban Mobility
• A Temporal Variational Model for Story Generation
• Latent Diffeomorphic Dynamic Mode Decomposition
• Long-Term Individual Causal Effect Estimation via Identifiable Latent Representation Learning
• Working Memory Graphs
• E4GEN: Event-level Explainable Extreme-Enhanced Time-series Generation
• Learning Disentangled Representations of Video with Missing Data
• Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics
• Disentangled Recurrent Wasserstein Autoencoder
• Improving Fair Predictions Using Variational Inference In Causal Models
• Learning to Set Waypoints for Audio-Visual Navigation
• A survey on Variational Autoencoders from a GreenAI perspective
• Improving Sequential Latent Variable Models with Autoregressive Flows
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• Relational State-Space Model for Stochastic Multi-Object Systems
• Preventing Posterior Collapse with delta-VAEs
• Goal-Directed Planning for Habituated Agents by Active Inference Using a Variational Recurrent Neural Network
• Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling

Related

• Regularized Sequential Latent Variable Models with Adversarial Neural Networks
• Sequential Neural Models with Stochastic Layers
• Z-Forcing: Training Stochastic Recurrent Networks
• Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data
• Variable Computation in Recurrent Neural Networks