xLSTM: Extended Long Short-Term Memory

2024/05/07 by Maximilian Beck, Korbinian Pöppel, Beck, Maximilian +17 · 10 voices · 77 citations
Computer Science · #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2405.04517

Abstract

In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.

Cited by

• Pretraining Recurrent Networks without Recurrence
• The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory
• BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis
• A Minimal Interpretable Architecture for Zero-Shot Reconstruction of Dynamical Systems
• Effective Distillation to Hybrid xLSTM Architectures
• M²RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
• Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling
• Deep learning for the probabilistic prediction of semi-continuous hydrological variables – An application to streamflow prediction across CONUS
• Kinaema: a recurrent sequence model for memory and pose in motion
• Fast weight programming and linear transformers: from machine learning to neurobiology
• Log-Linear Attention
• Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
• Breaking the Memory Wall: Exact Analytical Differentiation via Tiled Operator-Space Evolution
• Why Are Linear RNNs More Parallelizable?
• What Matters in Deep Learning for Time Series Forecasting?
• Compressing Observation History into Agent Memory: Distilling Transformers into Recurrent Transformers
• Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting
• Detection of Lensed Gravitational Waves in the Millihertz Band Using Frequency-Domain Lensing Feature Extraction Network
• ALERT Open Dataset and Input-Size-Agnostic Vision Transformer for Driver Activity Recognition using IR-UWB
• GDKVM: Echocardiography Video Segmentation via Spatiotemporal Key-Value Memory with Gated Delta Rule
• Privacy-Preserving Generative Modeling and Clinical Validation of Longitudinal Health Records for Chronic Disease
• Modèles de Fondation et Ajustement : Vers une Nouvelle Génération de Modèles pour la Prévision des Séries Temporelles
• On the Origin of Algorithmic Progress in AI
• Selective Rotary Position Embedding
• xLSTM-PINN: Memory-Gated Spectral Remodeling for Physics-Informed Learning
• TM-UNet: Token-Memory Enhanced Sequential Modeling for Efficient Medical Image Segmentation
• DenoGrad: A Gradient-Based Framework for Data Refinement in Tabular and Time-Series Learning
• UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNs
• Leveraging unlabelled data for generalizable neural population decoding
• Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principle
• FlashEVA: Accelerating LLM inference via Efficient Attention
• X-TRACK: Physics-Aware xLSTM for Realistic Vehicle Trajectory Prediction
• Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification
• Hybrid Quantum-Classical Recurrent Neural Networks
• TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting
• Parallel BiLSTM-Transformer networks for forecasting chaotic dynamics
• EMTSF:Extraordinary Mixture of SOTA Models for Time Series Forecasting
• PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
• SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series
• Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?
• Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• Exploring Resolution-Wise Shared Attention in Hybrid Mamba-U-Nets for Improved Cross-Corpus Speech Enhancement
• Tapered Language Models
• Design Principles for Sequence Models via Coefficient Dynamics
• Task-Level Insights from Eigenvalues across Sequence Models
• gLSTM: Mitigating Over-Squashing by Increasing Storage Capacity
• New Machine Learning Approaches for Intrusion Detection in ADS-B
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems
• XG-Attention-WGAN PIC: Utilizing XGboost-Attention-WGAN for Photonics Integrated Circuit Design
• VRWKV-Editor: Reducing quadratic complexity in transformer-based video editing
• TTT3R: 3D Reconstruction as Test-Time Training
• How Effective Are Time-Series Models for Precipitation Nowcasting? A Comprehensive Benchmark for GNSS-based Precipitation Nowcasting
• A Unifying Framework for Parallelizing Sequential Models with Linear Dynamical Systems
• Mamba Modulation: On the Length Generalization of Mamba
• On the Importance of Multistability for Horizon Generalization in Reinforcement Learning
• Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
• An overview of neural architectures for self-supervised audio representation learning from masked spectrograms
• Neural Network-Driven Direct CBCT-Based Dose Calculation for Head-and-Neck Proton Treatment Planning
• BenchECG and xECG: a benchmark and baseline for ECG foundation models
• Elucidating the Design Space of Decay in Linear Attention
• Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM
• Towards Multi-Aspect Diversification of News Recommendations Using Neuro-Symbolic AI for Individual and Societal Benefit
• StoxLSTM: A Stochastic Extended Long Short-Term Memory Network for Time Series Forecasting
• REVELIO -- Universal Multimodal Task Load Estimation for Cross-Domain Generalization
• Masked Autoencoder Pretraining and BiXLSTM ResNet Architecture for PET/CT Tumor Segmentation
• Yet Unnoticed in LSTM: Binary Tree Based Input Reordering, Weight Regularization, and Gate Nonlinearization
• HiSTM: Hierarchical Spatiotemporal Mamba for Cellular Traffic Forecasting
• FlowState: Sampling Rate Invariant Time Series Forecasting
• Minimal Convolutional RNNs Accelerate Spatiotemporal Learning
• JustDense: Just using Dense instead of Sequence Mixer for Time Series analysis
• TriP-LLM: A Tri-Branch Patch-wise Large Language Model Framework for Time-Series Anomaly Detection
• M-Net: MRI Brain Tumor Sequential Segmentation Network via Mesh-Cast
• Directly Learning Stock Trading Strategies Through Profit Guided Loss Functions
• Efficient Attention Mechanisms for Large Language Models: A Survey
• Long short-term memory [wikipedia]

Discussions

• xLSTM: Extended Long Short-Term Memory [hn, 197 points, 73 comments]
• XLSTM helps with the parallelizable thing arxiv.org/abs/2405.04517 I suspect the memory issues and compute scaling with sequence lengths will motivate some large scale model with these soon. Probably [bsky, 3 points, 0 comments]
• How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? #ML #AI #LSTM [bsky, 2 points, 0 comments]
• Interesting to see how far LSTM ideas can be pushed to make make them competitive with transformers and state space models. Proud of our paper and all my co-authors. arxiv.org/abs/2405.04517 [bsky, 1 points, 1 comments]
• e.g., I really want to talk to an xLSTM arxiv.org/abs/2405.04517 [bsky, 1 points, 0 comments]
• xLSTM is a combination of transformer technology and long-term memory. The result is an architecture that performs better in terms of performance and scalability than the transformers currently in use [bsky, 0 points, 0 comments]
• XLSTM: Extended Long Short-Term Memory (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• 2/2 xLSTM: Extended Long Short-Term Memory by Maximilian Beck, Andreas Auer, Günter Klambauer, Korbinian Pöppel, Oleksandra Prudnikova, Johannes Brandstetter, Markus Spanring, Michael Kopp, Sepp Hochr [bsky, 0 points, 0 comments]
• Strong Schmidhuber vibes incoming 🧐 (arxiv.org/abs/2405.04517) [bsky, 0 points, 0 comments]
• 夏休みでやるプロジェクトのためにLSTMについて調べたらこんなもの出てきた これは面白いが自分のプロジェクトに使えないかも arxiv.org/abs/2405.04517 [bsky, 0 points, 0 comments]

Related

• ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time
• xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers
• Grow and Prune Compact, Fast, and Accurate LSTMs