Scalable Gradients for Stochastic Differential Equations

2020/01/05 by Li, Xuechen, Wong, Ting-Kam Leonard, Chen, Ricky T. Q. +1 · 27 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Numerical Analysis (math.NA)

paper · doi:10.48550/arxiv.2001.01328

Abstract

The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations. We generalize this method to stochastic differential equations, allowing time-efficient and constant-memory computation of gradients with high-order adaptive solvers. Specifically, we derive a stochastic differential equation whose solution is the gradient, a memory-efficient algorithm for caching noise, and conditions under which numerical solutions converge. In addition, we combine our method with gradient-based stochastic variational inference for latent stochastic differential equations. We use our method to fit stochastic dynamics defined by neural networks, achieving competitive performance on a 50-dimensional motion capture dataset.

Cited by

• Diffusion differentiable resampling
• Self-Supervised Dynamical System Representations for Physiological Time-Series
• Integrating LSTM Networks with Neural Levy Processes for Financial Forecasting
• Sparse-to-Field Reconstruction via Stochastic Neural Dynamic Mode Decomposition
• Generative Modeling of Clinical Time Series via Latent Stochastic Differential Equations
• Curly Flow Matching for Learning Non-gradient Field Dynamics
• Neural Stochastic Flows: Solver-Free Modelling and Inference for SDE Solutions
• Multi-Marginal Schrödinger Bridge Matching
• Towards Fast Coarse-graining and Equation Discovery with Foundation Inference Models
• Efficient Inference for Coupled Hidden Markov Models in Continuous Time and Discrete Space
• Deep Neural Networks Inspired by Differential Equations
• Exact Solutions to the Quantum Schrödinger Bridge Problem
• RegVelo: Gene-regulatory-informed dynamics of single cells
• Diffusion Bridge Variational Inference for Deep Gaussian Processes
• Dynamic Gene Regulatory Network Inference with Interpretable, Biophysically-Motivated Neural ODEs
• Modeling Irregular Astronomical Time Series with Neural Stochastic Delay Differential Equations
• TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification
• Quantum neural ordinary and partial differential equations
• Reversible Deep Equilibrium Models
• Neural Stochastic Differential Equations on Compact State-Spaces
• Uncertainty Propagation Networks for Neural Ordinary Differential Equations
• A High-order Backpropagation Algorithm for Neural Stochastic Differential Equation Model
• Learning with springs and sticks
• STDiff: A State Transition Diffusion Framework for Time Series Imputation in Industrial Systems
• Multi-Marginal Stochastic Flow Matching for High-Dimensional Snapshot Data at Irregular Time Points
• The Vanishing Gradient Problem for Stiff Neural Differential Equations
• WeightFlow: Learning Stochastic Dynamics via Evolving Weight of Neural Network

Related

• An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations
• Learning Stochastic Differential Equations With Gaussian Processes\n Without Gradient Matching
• Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!
• Robust and Scalable SDE Learning: A Functional Perspective
• Amortized Reparametrization: Efficient and Scalable Variational Inference for Latent SDEs