ParallelTime: Dynamically Weighting the Balance of Short- and Long-Term Temporal Dependencies

2025/07/18 by Itay Katav, Katav, Itay, Aryeh Kontorovich +1
Computer Science · Physics and Astronomy · Psychology · #Complex Network Analysis Techniques #Data Visualization and Analytics #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mental Health Research Topics

paper · pdf · doi:10.48550/arxiv.2507.13998

openalex publication_date 2025/07/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Modern multivariate time series forecasting primarily relies on two architectures: the Transformer with attention mechanism and Mamba. In natural language processing, an approach has been used that combines local window attention for capturing short-term dependencies and Mamba for capturing long-term dependencies, with their outputs averaged to assign equal weight to both. We find that for time-series forecasting tasks, assigning equal weight to long-term and short-term dependencies is not optimal. To mitigate this, we propose a dynamic weighting mechanism, ParallelTime Weighter, which calculates interdependent weights for long-term and short-term dependencies for each token based on the input and the model's knowledge. Furthermore, we introduce the ParallelTime architecture, which incorporates the ParallelTime Weighter mechanism to deliver state-of-the-art performance across diverse benchmarks. Our architecture demonstrates robustness, achieves lower FLOPs, requires fewer parameters, scales effectively to longer prediction horizons, and significantly outperforms existing methods. These advances highlight a promising path for future developments of parallel Attention-Mamba in time series forecasting. The implementation is readily available at: \hrefhttps://github.com/itay1551/ParallelTimeGitHub.

Citations

• Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
• An Empirical Study of Mamba-based Language Models
• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
• Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysis
• Is Mamba Effective for Time Series Forecasting?
• MOMENT: A Family of Open Time-series Foundation Models
• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
• Vision Transformers Need Registers
• One Fits All:Power General Time Series Analysis by Pretrained LM
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis
• Simplified State Space Layers for Sequence Modeling
• Are Transformers Effective for Time Series Forecasting?
• FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting
• Efficiently Modeling Long Sequences with Structured State Spaces
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Root Mean Square Layer Normalization
• RobustTrend: A Huber Loss with a Combined First and Second Order Difference Regularization for Time Series Trend Filtering
• N-BEATS: Neural basis expansion analysis for interpretable time series forecasting
• Squeeze-and-Excitation Networks
• Attention Is All You Need
• Layer Normalization
• Deep Residual Learning for Image Recognition
• An Introduction to Convolutional Neural Networks
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting
• TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting
• MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal and Channel Mixing
• FMamba: Mamba based on Fast-attention for Multivariate Time-series Forecasting
• Timer-XL: Long-Context Transformers for Unified Time Series Forecasting