Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies

2019/01/22 by Sarath Chandar, Chinnadhurai Sankar, Chandar, Sarath +7 · 6 citations
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Parallel Computing and Optimization Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1902.06704

openalex publication_date 2019/01/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Modelling long-term dependencies is a challenge for recurrent neural networks. This is primarily due to the fact that gradients vanish during training, as the sequence length increases. Gradients can be attenuated by transition operators and are attenuated or dropped by activation functions. Canonical architectures like LSTM alleviate this issue by skipping information through a memory mechanism. We propose a new recurrent architecture (Non-saturating Recurrent Unit; NRU) that relies on a memory mechanism but forgoes both saturating activation functions and saturating gates, in order to further alleviate vanishing gradients. In a series of synthetic and real world tasks, we demonstrate that the proposed model is the only model that performs among the top 2 models across all tasks with and without long-term dependencies, when compared against a range of other architectures.

Citations

• Categorical Reparameterization with Gumbel-Softmax
• Memory Augmented Neural Networks with Wormhole Connections
• Can recurrent neural networks warp time?

Cited by

• Greedy dynamical meta-learning
• QL-LSTM: A Parameter-Efficient LSTM for Stable Long-Sequence Modeling
• Parallel Delayed Memory Units for Enhanced Temporal Modeling in Biomedical and Bioacoustic Signal Analysis
• Improving the Gating Mechanism of Recurrent Neural Networks
• A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions
• Non-normal Recurrent Neural Network (nnRNN): learning long time\n dependencies while improving expressivity with transient dynamics

Related

• Learning Long Term Dependencies via Fourier Recurrent Units
• Recurrent neural networks: vanishing and exploding gradients are not the end of the story
• Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey
• Gating Revisited: Deep Multi-layer RNNs That Can Be Trained
• Can recurrent neural networks warp time?