Latent Neural Differential Equations for Video Generation

2020/11/07 by Cade Gordon, Gordon, Cade, Natalie Parde +1 · 1 citation
Computer Science · #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #cs.CV

paper · pdf · doi:10.48550/arxiv.2011.03864

openalex publication_date 2020/11/07 · arxiv created 2021/05/11 · arxiv updated 2021/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Generative Adversarial Networks have recently shown promise for video generation, building off of the success of image generation while also addressing a new challenge: time. Although time was analyzed in some early work, the literature has not adequately grown with temporal modeling developments. We study the effects of Neural Differential Equations to model the temporal dynamics of video generation. The paradigm of Neural Differential Equations presents many theoretical strengths including the first continuous representation of time within video generation. In order to address the effects of Neural Differential Equations, we investigate how changes in temporal models affect generated video quality. Our results give support to the usage of Neural Differential Equations as a simple replacement for older temporal generators. While keeping run times similar and decreasing parameter count, we produce a new state-of-the-art model in 64×64 pixel unconditional video generation, with an Inception Score of 15.20.

Citations

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Long Short-Term Memory
• Unsupervised Learning of Video Representations using LSTMs
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Stochastic Video Generation with a Learned Prior
• Adversarial Video Generation on Complex Datasets
• High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
• Transformation-based Adversarial Video Prediction on Large-Scale Data

Cited by

• Physically Plausible Multi-System Trajectory Generation and Symmetry Discovery

Related

• Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation
• Continuous-Time Video Generation via Learning Motion Dynamics with Neural ODE
• Exploiting Inductive Biases in Video Modeling through Neural CDEs
• Generative Video Bi-flow
• Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks