Fourier-mixed window attention for efficient and robust long sequence time-series forecasting

2023/07/02 by Nhat Thanh Tran, Jack Xin, Tran, Nhat Thanh +1 · 2 citations
Computer Science · Decision Sciences · Mathematics · #Algorithm #Artificial intelligence #Computer science #Econometrics #Machine learning #Mathematics #Sequence (biology) #Series (stratigraphy) #Statistical and numerical algorithms #Stock Market Forecasting Methods #Time Series Analysis and Forecasting #Time sequence #Time series #Window (computing)

paper · pdf · doi:10.3389/fams.2025.1600136

published in Frontiers in Applied Mathematics and Statistics 11 (Frontiers Media)

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/23

Abstract

We study a fast local-global window-based attention method to accelerate Informer for long sequence time-series forecasting (LSTF) in a robust manner. While window attention being local is a considerable computational saving, it lacks the ability to capture global token information which is compensated by a subsequent Fourier transform block. Our method, named FWin, does not rely on query sparsity hypothesis and an empirical approximation underlying the ProbSparse attention of Informer. Experiments on univariate and multivariate datasets show that FWin transformers improve the overall prediction accuracies of Informer while accelerating its inference speeds by 1.6 to 2 times. On strongly non-stationary data (power grid and dengue disease data), FWin outperforms Informer and recent SOTAs thereby demonstrating its superior robustness . We give mathematical definition of FWin attention, and prove its equivalency to the canonical full attention under the block diagonal invertibility (BDI) condition of the attention matrix. The BDI is verified to hold with high probability on benchmark datasets experimentally.

Citations

• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
• FNet: Mixing Tokens with Fourier Transforms
• MLP-Mixer: An all-MLP Architecture for Vision
• Attention Is All You Need
• On Estimation of a Probability Density Function and Mode
• Remarks on Some Nonparametric Estimates of a Density Function
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• A sufficient condition for nonvanishing of determinants

Cited by

• Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units
• Filter then Attend: Improving attention-based Time Series Forecasting with Spectral Filtering

Related

• Revisiting Attention for Multivariate Time Series Forecasting
• FRWKV:Frequency-Domain Linear Attention for Long-Term Time Series Forecasting
• Attention as Robust Representation for Time Series Forecasting
• Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecasting
• Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting