Do RNN and LSTM have Long Memory?

2020/06/06 by Zhao, Jingyu, Huang, Feiqing, Lv, Jia +4 · 9 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2006.03860

Abstract

The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications. With its success and drawbacks in mind, this paper raises the question - do RNN and LSTM have long memory? We answer it partially by proving that RNN and LSTM do not have long memory from a statistical perspective. A new definition for long memory networks is further introduced, and it requires the model weights to decay at a polynomial rate. To verify our theory, we convert RNN and LSTM into long memory networks by making a minimal modification, and their superiority is illustrated in modeling long-term dependence of various datasets.

Cited by

• Deep Generative Spatiotemporal Engression for Probabilistic Forecasting of Epidemics
• QL-LSTM: A Parameter-Efficient LSTM for Stable Long-Sequence Modeling
• TimeGazer: Temporal Modeling of Predictive Gaze Stabilization for AR Interaction
• Machines in the Margins: A Systematic Review of Automated Content Generation for Wikipedia
• Leveraging IndoBERT and DistilBERT for Indonesian Emotion Classification in E-Commerce Reviews
• Neural ARFIMA model for forecasting BRIC exchange rates with long memory
• Deep Learning-based Animal Behavior Analysis: Insights from Mouse Chronic Pain Models
• Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing
• Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by nano-GPT

Related

• Learning Over Long Time Lags
• Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks
• Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey
• Long Short-term Memory RNN
• A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures