Were RNNs All We Needed?

2024/10/02 by Leo Feng, Feng, Leo, Frederick Tung +8 · 9 voices · 10 citations
Nursing · #Nursing Education, Practice, and Leadership #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2410.01201

openalex publication_date 2024/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers - particularly with respect to sequence length - have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively. In this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers.

Cited by

• Pretraining Recurrent Networks without Recurrence
• Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation
• Transformers without Normalization
• EMTSF:Extraordinary Mixture of SOTA Models for Time Series Forecasting
• Microstructure sensitive recurrent neural network surrogate model of crystal plasticity
• On the Importance of Multistability for Horizon Generalization in Reinforcement Learning
• Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
• Neuromorphic Intelligence
• Minimal Convolutional RNNs Accelerate Spatiotemporal Learning
• Data-Driven Greenhouse Climate Regulation in Lettuce Cultivation Using BiLSTM and GRU Predictive Control

Discussions

• Were RNNs all we needed? [hn, 520 points, 260 comments]
• #AdventOfPapers 2: Were RNNs all we needed? (rec from @vn-ml.bsky.social) This is partly a history of machine learning for sequences (tl;dr RNNs had vanishing+exploding gradients, then LSTMs and GRUs [bsky, 5 points, 1 comments]
• More and more research indicates that transformer models might not be the ultimate solution after all. Here is a paper to revive older rnn types: arxiv.org/abs/2410.01201 #deeplearning [bsky, 2 points, 0 comments]
• Will Transformers prove to be a 'transitional form' in the future AI fossil record? :P [bsky, 2 points, 0 comments]
• Digging into state space models and just came across arxiv.org/abs/2410.01201. Looks neat! I really like the work of revisiting older RNN architectures when you can jettison backprop through time [bsky, 1 points, 1 comments]
• Were RNNs all we needed? https://arxiv.org/abs/2410.01201 [bsky, 0 points, 0 comments]
• #minGRU seems interesting. Need to try it out. arxiv.org/abs/2410.01201 [bsky, 0 points, 0 comments]
• Were RNNs all we needed? https://arxiv.org/abs/2410.01201 [bsky, 0 points, 0 comments]
• Were RNNs All We Needed? [bsky, 0 points, 0 comments]

Related

• RWKV: Reinventing RNNs for the Transformer Era
• On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• Universal Transformers
• Long Distance Relationships without Time Travel: Boosting the Performance of a Sparse Predictive Autoencoder in Sequence Modeling