Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting

2025/09/25 by Jiayi Zhu, Liang, Zida, Zhu, Jiayi +2 · 1 citation
Computer Science · Decision Sciences · #FOS: Computer and information sciences #Forecasting Techniques and Applications #Machine Learning (cs.LG) #Stock Market Forecasting Methods #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2509.20942

openalex publication_date 2025/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer-based architectures achieved high performance in natural language processing and computer vision, yet many studies have shown that they have not demonstrated a clear advantage in time series forecasting and even underperform simple linear baselines in some cases. However, most of these studies have not thoroughly explored the reasons behind the failure of transformers. To better understand time-series transformers(TST), we designed a series of experiments, progressively modifying transformers into MLPs to investigate the impact of the attention mechanism. Surprisingly, transformer blocks often degenerate into simple MLPs in existing time-series transformers. We designed a interpretable dataset to investigate the reasons behind the failure of the attention mechanism and revealed that the attention mechanism is not working in the expected way. We theoretically analyzed the reasons behind this phenomenon, demonstrating that the current embedding methods fail to allow transformers to function in a well-structured latent space, and further analyzed the deeper underlying causes of the failure of embedding.

Citations

• RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting
• Are Self-Attentions Effective for Time Series Forecasting?
• Chronos: Learning the Language of Time Series
• Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling
• SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention
• Unified Training of Universal Time Series Forecasting Transformers
• Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting
• Timer: Generative Pre-trained Transformers Are Large Time Series Models
• Learning to Embed Time Series Patches Independently
• A decoder-only foundation model for time-series forecasting
• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
• CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Are Transformers Effective for Time Series Forecasting?
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Attention Is All You Need
• Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks
• Greedy function approximation: A gradient boosting machine.
• Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting

Cited by

• NeuroSketch: An Effective Framework for Neural Decoding via Systematic Architectural Optimization
• A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models

Related

• Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
• Why Do Transformers Fail to Forecast Time Series In-Context?
• Recency Biased Causal Attention for Time-series Forecasting
• Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
• Transformers predicting the future. Applying attention in next-frame and time series forecasting