A Critical Review of Recurrent Neural Networks for Sequence Learning

2015/05/29 by Zachary C. Lipton, Lipton, Zachary C., John Berkowitz +3 · 1 voice · 79 citations
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1506.00019

openalex publication_date 2015/05/29 · arxiv created 2015/10/17 · arxiv updated 2015/10/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models. When appropriate, we reconcile conflicting notation and nomenclature. Our goal is to provide a self-contained explication of the state of the art together with a historical perspective and references to primary research.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Learning to Execute
• Efficient Estimation of Word Representations in Vector Space
• Finding Structure in Time
• A Novel Connectionist System for Unconstrained Handwriting Recognition
• Sequence to Sequence Learning with Neural Networks
• word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
• Bidirectional recurrent neural networks
• Learning to Forget: Continual Prediction with LSTM
• ImageNet classification with deep convolutional neural networks
• Deeply-Supervised Nets
• Sequence to Sequence -- Video to Text

Cited by

• EIR: Enhanced Image Representations for Medical Report Generation
• Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI
• A Lightweight Approach to Detection of AI-Generated Texts Using Stylometric Features
• Predicting and forecasting reactivity and flux using long short-term memory models in pebble bed reactors during run-in
• Fast surrogate modelling of EIT in atomic quantum systems using LSTM neural networks
• Moving or Predicting? RoleAware-MAPP: A Role-Aware Transformer Framework for Movable Antenna Position Prediction to Secure Wireless Communications
• SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series
• Variability Aware Recursive Neural Network (VARNN): A Residual-Memory Model for Capturing Temporal Deviation in Sequence Regression Modeling
• QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
• Overcoming the curse of dimensionality: Enabling multi-layer photon transport with recurrent neural network
• Enhancing Linear Attention with Residual Learning
• Learning to Diagnose with LSTM Recurrent Neural Networks
• Exploiting Review Neighbors for Contextualized Helpfulness Prediction
• Sleep Disorder Diagnosis Using EEG Signals and LSTM Deep Learning Method
• Notes on Deep Learning for NLP
• A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
• Parkinson's Disease Diagnosis Using Deep Learning
• AI-Based Detection, Classification and Prediction/Prognosis in Medical Imaging: Towards Radiophenomics
• Addressing Two Problems in Deep Knowledge Tracing via Prediction-Consistent Regularization
• DETECT: Deep Trajectory Clustering for Mobility-Behavior Analysis
• TIES: Temporal Interaction Embeddings For Enhancing Social Media Integrity At Facebook
• OadTR: Online Action Detection with Transformers
• Adaptive Neural Quantum States: A Recurrent Neural Network Perspective
• Recurrent Neural Networks for anomaly detection in the Post-Mortem time series of LHC superconducting magnets
• A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions
• Deep Trustworthy Knowledge Tracing
• Recurrent Control Nets for Deep Reinforcement Learning
• When Speed meets Accuracy: an Efficient and Effective Graph Model for Temporal Link Prediction
• R2N2: Residual Recurrent Neural Networks for Multivariate Time Series Forecasting
• A Conceptual Development of Quench Prediction App build on LSTM and ELQA framework
• General Backpropagation Algorithm for Training Second-order Neural Networks
• Connecting Software Metrics across Versions to Predict Defects
• Predictive process mining by network of classifiers and clusterers: the PEDF model
• Characterization and recognition of handwritten digits using Julia
• Ground Reaction Force Estimation via Time-aware Knowledge Distillation
• A CNN-LSTM Quantifier for Single Access Point CSI Indoor Localization
• Sentiment Analysis in Learning Management Systems Understanding Student Feedback at Scale
• Robust Named Entity Recognition in Idiosyncratic Domains
• Sleep Stage Classification Based on Multi-level Feature Learning and Recurrent Neural Networks via Wearable Device
• Sequence-Aware Recommender Systems
• Multivariate, Multistep Forecasting, Reconstruction and Feature Selection of Ocean Waves via Recurrent and Sequence-to-Sequence Networks
• Hidden Markov Chains, Entropic Forward-Backward, and Part-Of-Speech Tagging
• Phenotyping of Clinical Time Series with LSTM Recurrent Neural Networks
• Listening to Sounds of Silence for Speech Denoising
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Leveraging recurrence in neural network wavefunctions for large-scale simulations of Heisenberg antiferromagnets on the triangular lattice
• geoGAT: Graph Model Based on Attention Mechanism for Geographic Text Classification
• ChatGPT: Jack of all trades, master of none
• FlashForge: Ultra-Efficient Prefix-Aware Attention for LLM Decoding
• BGADAM: Boosting based Genetic-Evolutionary ADAM for Neural Network Optimization
• Neuro-Argumentative Learning with Case-Based Reasoning
• Dual-Agent Reinforcement Learning for Automated Feature Generation
• Neural Language Generation: Formulation, Methods, and Evaluation
• SynDEc: A Synthetic Data Ecosystem
• Predicting Drug Responses by Propagating Interactions through Text-Enhanced Drug-Gene Networks
• Deep Learning Advances in Vision-Based Traffic Accident Anticipation: A Comprehensive Review of Methods, Datasets, and Future Directions
• Improving Context Modeling in Neural Topic Segmentation
• Probabilistic Modeling for Novelty Detection with Applications to Fraud Identification
• State of the Art for Solar and Wind Energy-Forecasting Methods for Sustainable Grid Integration
• PointExplainer: Towards Transparent Parkinson's Disease Diagnosis
• Identifying the fragment structure of the organic compounds by deeply learning the original NMR data
• Knowledge Tracing: A Survey
• Incremental Sequence Learning
• DeepDPM: Dynamic Population Mapping via Deep Neural Network
• Augmentation Scheme for Dealing with Imbalanced Network Traffic Classification Using Deep Learning
• Training LSTM Networks With Resistive Cross-Point Devices
• Causally Driven Incremental Multi Touch Attribution Using a Recurrent Neural Network
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• Multimodal Data Fusion based on the Global Workspace Theory
• A GAN-based Approach for Mitigating Inference Attacks in Smart Home Environment
• A Deep Prediction Network for Understanding Advertiser Intent and Satisfaction
• Bionoi: A Voronoi Diagram-Based Representation of Ligand-Binding Sites in Proteins for Machine Learning Applications
• Beam Search with Bidirectional Strategies for Neural Response Generation
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
• Construction of a far-ultraviolet all-sky map from an incomplete survey: application of a deep learning algorithm
• Artificial intelligence and deep learning algorithms for epigenetic sequence analysis: A review for epigeneticists and AI experts
• DR-RNN: A deep residual recurrent neural network for model reduction
• Automatic Model Selection for Neural Networks
• Sequence-based dynamic handwriting analysis for Parkinson’s disease detection with one-dimensional convolutions and BiGRUs
• Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks

Discussions

• A Critical Review of Recurrent Neural Networks for Sequence Learning [hn, 1 points, 0 comments]

Related

• Recurrent Neural Networks (RNNs): A gentle Introduction and Overview
• Learning The Sequential Temporal Information with Recurrent Neural Networks
• A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
• Seq2Biseq: Bidirectional Output-wise Recurrent Neural Networks for Sequence Modelling
• Generating Sequences With Recurrent Neural Networks