MinimalRNN: Toward More Interpretable and Trainable Recurrent Neural Networks

2017/11/18 by Minmin Chen, Chen, Minmin · 3 citations
Computer Science · Mathematics · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications #Topic Modeling #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1711.06788

Presented at NIPS 2017 Symposium on Interpretable Machine Learning

openalex publication_date 2017/11/18 · arxiv created 2018/06/20 · arxiv updated 2018/06/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce MinimalRNN, a new recurrent neural network architecture that achieves comparable performance as the popular gated RNNs with a simplified structure. It employs minimal updates within RNN, which not only leads to efficient learning and testing but more importantly better interpretability and trainability. We demonstrate that by endorsing the more restrictive update rule, MinimalRNN learns disentangled RNN states. We further examine the learning dynamics of different RNN structures using input-output Jacobians, and show that MinimalRNN is able to capture longer range dependencies than existing RNN architectures.

Citations

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Exploring the Limits of Language Modeling
• Session-based Recommendations with Recurrent Neural Networks
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation

Cited by

• Context-aware Sparse Spatiotemporal Learning for Event-based Vision
• Transformer Model for Alzheimer's Disease Progression Prediction Using Longitudinal Visit Sequences
• Sparse Convolutional Recurrent Learning for Efficient Event-based Neuromorphic Object Detection

Related

• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• Minimal Gated Unit for Recurrent Neural Networks
• Network of Recurrent Neural Networks
• Rethinking Full Connectivity in Recurrent Neural Networks
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN