Learning Novel Transformer Architecture for Time-series Forecasting

2025/02/19 by Juyuan Zhang, Wei Zhu, Zhang, Juyuan +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2502.13721

openalex publication_date 2025/02/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the success of Transformer-based models in the time-series prediction (TSP) tasks, the existing Transformer architecture still face limitations and the literature lacks comprehensive explorations into alternative architectures. To address these challenges, we propose AutoFormer-TS, a novel framework that leverages a comprehensive search space for Transformer architectures tailored to TSP tasks. Our framework introduces a differentiable neural architecture search (DNAS) method, AB-DARTS, which improves upon existing DNAS approaches by enhancing the identification of optimal operations within the architecture. AutoFormer-TS systematically explores alternative attention mechanisms, activation functions, and encoding operations, moving beyond the traditional Transformer design. Extensive experiments demonstrate that AutoFormer-TS consistently outperforms state-of-the-art baselines across various TSP benchmarks, achieving superior forecasting accuracy while maintaining reasonable training efficiency.

Related

• The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecasting
• Transformers predicting the future. Applying attention in next-frame and time series forecasting
• Recency Biased Causal Attention for Time-series Forecasting
• A Systematic Review for Transformer-based Long-term Series Forecasting
• Enhancing Transformer-based models for Long Sequence Time Series Forecasting via Structured Matrix