Customizing Sequence Generation with Multi-Task Dynamical Systems

2019/10/11 by Alex D. Bird, Bird, Alex, Christopher K. I. Williams +1
Computer Science · #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.1910.05026

openalex publication_date 2019/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application. In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent code z that specifies the customization to the individual data sequence. This enables style transfer, interpolation and morphing within generated sequences. We show the MTDS can improve predictions via latent code interpolation, and avoid the long-term performance degradation of standard RNN approaches.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Auto-Encoding Variational Bayes
• Unconstrained parametrizations for variance-covariance matrices
• Unsupervised Learning of Video Representations using LSTMs
• Variational Lossy Autoencoder
• DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks
• Phase-functioned neural networks for character control
• A Survey on Multi-Task Learning
• Few‐shot Learning of Homogeneous Human Locomotion Styles
• Multi-Task Time Series Analysis applied to Drug Response Modelling
• Disentangled State Space Representations
• Disentangled Sequential Autoencoder
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data

Related

• Variational Hyper RNN for Sequence Modeling
• Design Principles for Sequence Models via Coefficient Dynamics
• Hybrid Neural Models For Sequence Modelling: The Best Of Three Worlds
• Real-time interactive sequence generation and control with Recurrent\n Neural Network ensembles
• Twin Networks: Matching the Future for Sequence Generation