UnICORNN: A recurrent model for learning very long time dependencies

2021/03/09 by T. Konstantin Rusch, Siddhartha Mishra, Rusch, T. Konstantin +1 · 10 citations
Computer Science · Mathematics · #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #math.DS #stat.ML

paper · pdf · doi:10.48550/arxiv.2103.05487

arxiv created 2021/06/10 · arxiv updated 2021/08/19

Abstract

The design of recurrent neural networks (RNNs) to accurately process sequential inputs with long-time dependencies is very challenging on account of the exploding and vanishing gradient problem. To overcome this, we propose a novel RNN architecture which is based on a structure preserving discretization of a Hamiltonian system of second-order ordinary differential equations that models networks of oscillators. The resulting RNN is fast, invertible (in time), memory efficient and we derive rigorous bounds on the hidden state gradients to prove the mitigation of the exploding and vanishing gradient problem. A suite of experiments are presented to demonstrate that the proposed RNN provides state of the art performance on a variety of learning tasks with (very) long-time dependencies.

Cited by

• Upper Approximation Bounds for Neural Oscillators
• Stuart-Landau Oscillatory Graph Neural Network
• Hankel Singular Value Regularization for Highly Compressible State Space Models
• ASecond-Order SpikingSSM for Wearables
• TimeGazer: Temporal Modeling of Predictive Gaze Stabilization for AR Interaction
• Myosotis: structured computation for attention like layer
• Over-squashing in Spatiotemporal Graph Neural Networks
• Learning long range dependencies through time reversal symmetry breaking
• Learning to Dissipate Energy in Oscillatory State-Space Models
• Block-Biased Mamba for Long-Range Sequence Processing

Related

• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Recurrent Neural Networks in the Eye of Differential Equations
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems