SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

2024/02/15 by Romain Ilbert, Ambroise Odonnat, Ilbert, Romain +11 · 12 citations
Computer Science · Decision Sciences · #FOS: Computer and information sciences #Forecasting Techniques and Applications #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stock Market Forecasting Methods #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2402.10198

openalex publication_date 2024/02/15 · openalex created_date 2024/02/18 · openalex updated_date 2026/07/28

Abstract

Transformer-based architectures achieved breakthrough performance in natural language processing and computer vision, yet they remain inferior to simpler linear baselines in multivariate long-term forecasting. To better understand this phenomenon, we start by studying a toy linear forecasting problem for which we show that transformers are incapable of converging to their true solution despite their high expressive power. We further identify the attention of transformers as being responsible for this low generalization capacity. Building upon this insight, we propose a shallow lightweight transformer model that successfully escapes bad local minima when optimized with sharpness-aware optimization. We empirically demonstrate that this result extends to all commonly used real-world multivariate time series datasets. In particular, SAMformer surpasses current state-of-the-art methods and is on par with the biggest foundation model MOIRAI while having significantly fewer parameters. The code is available at https://github.com/romilbert/samformer.

Cited by

• IdealTSF: Can Non-Ideal Data Contribute to Enhancing the Performance of Time Series Forecasting Models?
• AlphaCast: A Human Wisdom-LLM Intelligence Co-Reasoning Framework for Interactive Time Series Forecasting
• Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis
• MCANet: A Coherent Multimodal Collaborative Attention Network for Advanced Modulation Recognition in Adverse Noisy Environments
• Why Cannot Neural Networks Master Extrapolation? Insights from Physical Laws
• Foundation vs. Specialized Models: Evaluating Catastrophic Forgetting in Continual Time Series Forecasting
• How Foundational are Foundation Models for Time Series Forecasting?
• Channel, Trend and Periodic-Wise Representation Learning for Multivariate Long-term Time Series Forecasting
• Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting
• Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Models
• One task to rule them all: A closer look at traffic classification generalizability
• Evaluation of a Foundational Model and Stochastic Models for Forecasting Sporadic or Spiky Production Outages of High-Performance Machine Learning Services

Related

• LATST: Are Transformers Necessarily Complex for Time-Series Forecasting
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
• Are Transformers Effective for Time Series Forecasting?
• Are Transformers Effective for Time Series Forecasting?