Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

2022/06/08 by Amin Shabani, Shabani, Amin, Amir H. Abdi +5 · 11 citations
Computer Science · Decision Sciences · Economics, Econometrics and Finance · #Complex Systems and Time Series Analysis #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stock Market Forecasting Methods #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2206.04038

openalex publication_date 2022/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The performance of time series forecasting has recently been greatly improved by the introduction of transformers. In this paper, we propose a general multi-scale framework that can be applied to the state-of-the-art transformer-based time series forecasting models (FEDformer, Autoformer, etc.). By iteratively refining a forecasted time series at multiple scales with shared weights, introducing architecture adaptations, and a specially-designed normalization scheme, we are able to achieve significant performance improvements, from 5.5% to 38.5% across datasets and transformer architectures, with minimal additional computational overhead. Via detailed ablation studies, we demonstrate the effectiveness of each of our contributions across the architecture and methodology. Furthermore, our experiments on various public datasets demonstrate that the proposed improvements outperform their corresponding baseline counterparts. Our code is publicly available in https://github.com/BorealisAI/scaleformer.

Cited by

• Cisco Time Series Model Technical Report
• Denoising Refinement Diffusion Models for Simultaneous Generation of Multi-scale Mobile Network Traffic
• HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series
• InvDec: Inverted Decoder for Multivariate Time Series Forecasting with Separated Temporal and Variate Modeling
• MCANet: A Coherent Multimodal Collaborative Attention Network for Advanced Modulation Recognition in Adverse Noisy Environments
• DSAT-HD: Dual-Stream Adaptive Transformer with Hybrid Decomposition for Multivariate Time Series Forecasting
• CoSupFormer : A Contrastive Supervised learning approach for EEG signal Classification
• GluMind: Multimodal Parallel Attention and Knowledge Retention for Robust Cross-Population Blood Glucose Forecasting
• IEFS-GMB: Gradient Memory Bank-Guided Feature Selection Based on Information Entropy for EEG Classification of Neurological Disorders
• STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction
• Foundation Models for Clean Energy Forecasting: A Comprehensive Review

Related

• Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting
• MultiResFormer: Transformer with Adaptive Multi-Resolution Modeling for General Time Series Forecasting
• ETSformer: Exponential Smoothing Transformers for Time-series Forecasting
• Synthetic Time Series Forecasting with Transformer Architectures: Extensive Simulation Benchmarks
• The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecasting