Do RNN and LSTM have Long Memory?

2020/06/06 by Jingyu Zhao, Zhao, Jingyu, Feiqing Huang +11 · 40 citations
Computer Science · Decision Sciences · Mathematics · #Artificial intelligence #Artificial neural network #Computer science #FOS: Computer and information sciences #Long short term memory #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Neural Networks and Applications #Perspective (graphical) #Recurrent neural network #Stock Market Forecasting Methods #Time Series Analysis and Forecasting #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.03860

published in arXiv (Cornell University) (Cornell University) · Accepted by ICML 2020. Added references, experiments and acknowledgements

openalex publication_date 2020/06/06 · arxiv created 2020/06/10 · arxiv updated 2020/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications. With its success and drawbacks in mind, this paper raises the question - do RNN and LSTM have long memory? We answer it partially by proving that RNN and LSTM do not have long memory from a statistical perspective. A new definition for long memory networks is further introduced, and it requires the model weights to decay at a polynomial rate. To verify our theory, we convert RNN and LSTM into long memory networks by making a minimal modification, and their superiority is illustrated in modeling long-term dependence of various datasets.

Citations

• Long Short-Term Memory
• Machine Learning on Sequential Data Using a Recurrent Weighted Average

Cited by

• Deep Generative Spatiotemporal Engression for Probabilistic Forecasting of Epidemics
• QL-LSTM: A Parameter-Efficient LSTM for Stable Long-Sequence Modeling
• TimeGazer: Temporal Modeling of Predictive Gaze Stabilization for AR Interaction
• Machines in the Margins: A Systematic Review of Automated Content Generation for Wikipedia
• Leveraging IndoBERT and DistilBERT for Indonesian Emotion Classification in E-Commerce Reviews
• Neural ARFIMA model for forecasting BRIC exchange rates with long memory
• Deep Learning-based Animal Behavior Analysis: Insights from Mouse Chronic Pain Models
• Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing
• Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by nano-GPT
• Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions
• Deep Inertial Odometry with Accurate IMU Preintegration

Related

• Learning Over Long Time Lags
• Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks
• Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey
• Long Short-term Memory RNN
• A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures