Learning to Forget: Continual Prediction with LSTM

2000/10/01 by Felix A. Gers, Jürgen Schmidhuber, Fred Cummins · 185 citations
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #Neural Networks and Applications

paper · doi:10.1162/089976600300015015

openalex publication_date 2000/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.

Cited by

• G-PIFNN: A Generalizable Physics-informed Fourier Neural Network Framework for Electrical Circuits
• Tracing Footsteps of Similar Cities: Modeling Urban Economic Vitality with Dynamic Inter-City Graph Embeddings
• Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI
• MPCGNet: A Multiscale Feature Extraction and Progressive Feature Aggregation Network Using Coupling Gates for Polyp Segmentation
• Research progress of artificial intelligence empowered quantum communication and quantum sensing systems
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models
• MaGNet: A Mamba Dual-Hypergraph Network for Stock Prediction via Temporal-Causal and Global Relational Learning
• A Review of End-to-End Precipitation Prediction Using Remote Sensing Data: from Divination to Machine Learning
• Deep Learning Approach for Intelligent Named Entity Recognition of Cyber Security
• Towards Relevance and Sequence Modeling in Language Recognition
• Recurrence-Complete Frame-based Action Models
• Spiking Neural Network for Cross-Market Portfolio Optimization in Financial Markets: A Neuromorphic Computing Approach
• Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings
• Gait Recognition via Disentangled Representation Learning
• Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression
• Neural Network State-Space Estimators
• Interactive Video Object Segmentation Using Global and Local Transfer Modules
• Who invented deep residual learning?
• Toward a Semi-Supervised Learning Approach to Phylogenetic Estimation
• Learning to Search for MIMO Detection
• Complementary Matrix-Gated QKAN Fast-Weight Programmers for Quantum Dynamics Forecasting
• Brain-inspired wiring economics for artificial neural networks
• Sentiment Analysis on Bangla and Romanized Bangla Text (BRBT) using Deep Recurrent models
• Deviant Learning Algorithm: Learning Sparse Mismatch Representations through Time and Space
• Learning to Diagnose with LSTM Recurrent Neural Networks
• A Robust Deep Learning Approach for Automatic Classification of Seizures Against Non-seizures
• The unreasonable effectiveness of the forget gate
• Efficiently applying attention to sequential data with the Recurrent Discounted Attention unit
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
• A new deep learning-based approach for predicting the geothermal heat pump’s thermal power of a real bioclimatic house
• Artificial neural networks for neuroscientists: A primer
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• STFCN: Spatio-Temporal FCN for Semantic Video Segmentation
• Improved Classification of Nitrogen Stress Severity in Plants Under Combined Stress Conditions Using Spatio-Temporal Deep Learning Framework
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Dota 2 with Large Scale Deep Reinforcement Learning
• Mapping high-performance RNNs to in-memory neuromorphic chips
• Emulating Global 21 cm Cosmology Observations from the Lunar Far Side to Achieve Quick and Reliable Physical Constraints
• New Millennium AI and the Convergence of History
• FinCast: A Foundation Model for Financial Time-Series Forecasting
• DeepCP: Deep Learning Driven Cascade Prediction Based Autonomous Content Placement in Closed Social Network
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
• DeepTFP: Mobile Time Series Data Analytics based Traffic Flow Prediction
• Deep Recurrent Neural Network for Mobile Human Activity Recognition with High Throughput
• Uncertainty‐aware structural damage warning system using deep variational composite neural networks
• Deep Relevance Ranking Using Enhanced Document-Query Interactions
• Bridging LSTM Architecture and the Neural Dynamics during Reading
• Enhancing Chinese Intent Classification by Dynamically Integrating Character Features into Word Embeddings with Ensemble Techniques
• RUDDER: Return Decomposition for Delayed Rewards
• Using Chinese Glyphs for Named Entity Recognition
• Use of 1D-CNN for input data size reduction of LSTM in Hourly Rainfall-Runoff modeling
• A Machine Learning Model for Predicting Progressive Crack Extension based on DCPD Fatigue Data
• Improved spatial memory in a modular network mimicking the prefrontal-thalamo-hippocampal triangular circuit
• Transgender Community Sentiment Analysis from Social Media Data: A Natural Language Processing Approach
• The Vanishing Gradient Problem for Stiff Neural Differential Equations
• Long Short-Term Memory based Convolutional Recurrent Neural Networks for Large Vocabulary Speech Recognition
• Video Summarization with Long Short-term Memory
• A Simple LSTM model for Transition-based Dependency Parsing
• Multimodal deep learning for short-term stock volatility prediction
• Neural Decision Trees
• Tree-to-Sequence Attentional Neural Machine Translation
• Towards Playlist Generation Algorithms Using RNNs Trained on Within-Track Transitions
• Deep learning with long short-term memory networks for financial market predictions
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Teaching a Machine to Diagnose a Heart Disease; Beginning from digitizing scanned ECGs to detecting the Brugada Syndrome (BrS)
• Investigating the Effect of Segmentation Methods on Neural Model based Sentiment Analysis on Informal Short Texts in Turkish
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• Attentive Recurrent Comparators
• A Deep Network with Visual Text Composition Behavior
• Decoding Working Memory Load from EEG with LSTM Networks
• Incremental Adaptation of NMT for Professional Post-editors: A User Study
• Variational inference formulation for a model-free simulation of a dynamical system with unknown parameters by a recurrent neural network
• Deep Learning: Our Miraculous Year 1990-1991
• Long-Term Mobile Traffic Forecasting Using Deep Spatio-Temporal Neural Networks
• A deep hybrid learning model for customer repurchase behavior
• Natural Language Understanding with Distributed Representation
• Who Will Share My Image? Predicting the Content Diffusion Path in Online Social Networks
• Keyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized Embeddings
• Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency
• Time-series forecasting for nonlinear high-dimensional system using hybrid method combining autoencoder and multi-parallelized quantum long short-term memory and gated recurrent unit
• Multi-Decoder RNN Autoencoder Based on Variational Bayes Method
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• A Rolling Optimized Nonlinear Grey Bernoulli Model RONGBM(1,1) and application in predicting total COVID-19 infected cases
• Associative Long Short-Term Memory
• A Comparative Analysis of Forecasting Financial Time Series Using ARIMA, LSTM, and BiLSTM
• PRING: Rethinking Protein-Protein Interaction Prediction from Pairs to Graphs
• Astronomia ex machina: a history, primer and outlook on neural networks in astronomy
• Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by nano-GPT
• Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning
• End-to-End Large Portfolio Optimization for Variance Minimization with Neural Networks through Covariance Cleaning
• mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling
• LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
• Discovery of the Hidden State in Ionic Models Using a Domain-Specific Recurrent Neural Network
• Learning to Start for Sequence to Sequence Architecture
• Image segmentation via Cellular Automata
• Evaluating Sound Similarity Metrics for Differentiable, Iterative Sound-Matching
• Self-Adaptive Hierarchical Sentence Model
• An LSTM approach to Forecast Migration using Google Trends
• Evolino for recurrent support vector machines
• Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers
• My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013
• Spatial Attention as an Interface for Image Captioning Models
• Compensation Learning
• Macroeconomic forecasting with LSTM and mixed frequency time series data
• Learning User Intent from Action Sequences on Interactive Systems
• SECTOR: A Neural Model for Coherent Topic Segmentation and Classification
• Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis
• Forecasting Daily COVID-19 Related Calls in VA Health Care System: Predictive Model Development
• Deep Multimodal Image-Text Embeddings for Automatic Cross-Media Retrieval
• pLSTM: parallelizable Linear Source Transition Mark networks
• Data-driven Day Ahead Market Prices Forecasting: A Focus on Short Training Set Windows
• Advancing Exchange Rate Forecasting: Leveraging Machine Learning and AI for Enhanced Accuracy in Global Financial Markets
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Multi-agent Bayesian Deep Reinforcement Learning for Microgrid Energy Management under Communication Failures
• MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
• DeepDrawing: A Deep Learning Approach to Graph Drawing
• PowerNet: Neural Power Demand Forecasting in Smart Grid
• DNA-Level Splice Junction Prediction using Deep Recurrent Neural Networks
• How to Train Your Metamorphic Deep Neural Network
• Neural Emoji Recommendation in Dialogue Systems
• Phenotyping of Clinical Time Series with LSTM Recurrent Neural Networks
• Incorporating Discrete Translation Lexicons into Neural Machine Translation
• PipeTune: Pipeline Parallelism of Hyper and System Parameters Tuning for Deep Learning Clusters
• Deep Learning Weather Models for Subregional Ocean Forecasting: A Case Study on the Canary Current Upwelling System
• Counting in Language with RNNs
• Log-Linear RNNs: Towards Recurrent Neural Networks with Flexible Prior Knowledge
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
• Polyphonic Music Composition with LSTM Neural Networks and Reinforcement Learning
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• Deformation prediction of large-scale civil structures using spatiotemporal clustering and empirical mode decomposition-based long short-term memory network
• Hybrid Models for Financial Forecasting: Combining Econometric, Machine Learning, and Deep Learning Models
• Semantic Modelling with Long-Short-Term Memory for Information Retrieval
• Robust Bandit Learning with Imperfect Context
• Seeing in the dark with recurrent convolutional neural networks
• Disentangled Item Representation for Recommender Systems
• A Network Science Approach to Granular Time Series Segmentation
• Generating Image Sequence from Description with LSTM Conditional GAN
• Improving Aspect Term Extraction with Bidirectional Dependency Tree Representation
• Precision Highway for Ultra Low-Precision Quantization
• KernelOracle: Predicting the Linux Scheduler's Next Move with Deep Learning
• Time Series Data Cleaning with Regular and Irregular Time Intervals
• Surrogate Optimization of Deep Neural Networks for Groundwater Predictions
• Deep Architectures and Ensembles for Semantic Video Classification
• Gated Feedback Recurrent Neural Networks
• Dynamical principles of habituation across substrates and scales
• Disease Progression Modeling Workbench 360
• Generative Adversarial Network for Probabilistic Forecast of Random Dynamical System
• An LSTM Recurrent Network for Step Counting
• Gauravarora@HASOC-Dravidian-CodeMix-FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection
• Deep Learning applied to Road Traffic Speed forecasting
• Model predictive control design for dynamical systems learned by Long Short-Term Memory Networks
• Gated Recurrent Unit Based Acoustic Modeling with Future Context
• On Subquadratic Architectures: From Applications to Principles
• Improving LEO short-term orbit prediction using LSTM neural network
• Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
• Generating Images from Captions with Attention
• Recurrent Back-Projection Network for Video Super-Resolution
• Suspicious News Detection Using Micro Blog Text
• Impact of Medical Data Imprecision on Learning Results
• Hybrid Quantum Recurrent Neural Network For Remaining Useful Life Prediction
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• A review on the long short-term memory model
• Estimating the Conditional Forecast-Revision Scale in Sequential Models: Local-Smoothing Limits, Matched Models, and Cost--Accuracy Trade-offs
• Auto-Vectorizing TensorFlow Graphs: Jacobians, Auto-Batching And Beyond
• Regularized Flexible Activation Function Combinations for Deep Neural Networks
• Matching Questions and Answers in Dialogues from Online Forums
• Evaluating Time Series Models for Urban Wastewater Management: Predictive Performance, Model Complexity and Resilience
• Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries
• Benchmarking Deep Learning Models for Dense Event Classification of Offshore Wind Infrastructure in Sentinel-1 Time Series
• Convolutional neural networks for automatic meter reading
• Learning Short-Cut Connections for Object Counting
• Survey of Loss Augmented Knowledge Tracing
• Neural Classification of Malicious Scripts: A study with JavaScript and VBScript
• <b>Anomaly Detection for Railway Vehicle Equipment Using Condition Monitoring Data</b>
• Jet flavor classification in high-energy physics with deep neural networks
• A systematic review on deep learning architectures and applications
• Predictive control of blast furnace temperature in steelmaking with hybrid depth-infused quantum neural networks
• Smart Cities: A Survey on Data Management, Security, and Enabling Technologies
• Cross-Sectional Heterogeneity in LSTM Networks for Financial Time Series
• SRVP: Strong Recollection Video Prediction Model Using Attention-Based Spatiotemporal Correlation Fusion
• Partially stochastic deep learning with uncertainty quantification for model predictive heating control
• Agile Effort Estimation: Have We Solved the Problem Yet? Insights From a Replication Study

Related

• Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay
• Mitigating Catastrophic Forgetting in Long Short-Term Memory Networks
• Long Short-Term Memory
• A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
• Learning to Skip Ineffectual Recurrent Computations in LSTMs