Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention

2025/10/20 by Sun, Yinbo, Fang, Yuchen, Zhu, Zhibo +7 · 1 citation
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.21795

Abstract

The rapid advancement of time series foundation models (TSFMs) has been propelled by migrating architectures from language models. While existing TSFMs demonstrate impressive performance, their direct adoption of cross-domain architectures constrains effective capture of multiscale temporal dependencies inherent to time series data. This limitation becomes particularly pronounced during zero-shot transfer across datasets with divergent underlying patterns and sampling strategies. To address these challenges, we propose Hierarchical Interleaved Block Attention (HIBA) which employs hierarchical inter- and intra-block sparse attention to effectively capture multi-scale dependencies. Intra-block attention facilitates local information exchange, and inter-block attention operates across blocks to capture global temporal pattern interaction and dynamic evolution. Leveraging the HIBA architecture, we introduce Xihe, a scalable TSFM family spanning from an ultra-efficient 9.5M parameter configuration to high-capacity 1.5B variant. Evaluated on the comprehensive GIFT-Eval benchmark, our most compact Xihe-tiny model (9.5M) surpasses the majority of contemporary TSFMs, demonstrating remarkable parameter efficiency. More impressively, Xihe-max (1.5B) establishes new state-of-the-art zero-shot performance, surpassing previous best results by a substantial margin. This consistent performance excellence across the entire parameter spectrum provides compelling evidence for the exceptional generalization capabilities and architectural superiority of HIBA.

Citations

• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
• TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
• Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model
• Sundial: A Family of Highly Capable Time Series Foundation Models
• Towards Neural Scaling Laws for Time Series Foundation Models
• Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
• GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation
• Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts
• VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters
• The Llama 3 Herd of Models
• DAM: Towards A Foundation Model for Time Series Forecasting
• Toto: Time Series Optimized Transformer for Observability
• Chronos: Learning the Language of Time Series
• Unified Training of Universal Time Series Forecasting Transformers
• Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series
• A decoder-only foundation model for time-series forecasting
• Large Language Models Are Zero-Shot Time Series Forecasters
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
• TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Are Transformers Effective for Time Series Forecasting?
• Temporal Fusion Transformers for Interpretable Multi-horizon Time Series\n Forecasting
• Temporal Fusion Transformers for interpretable multi-horizon time series forecasting
• N-BEATS: Neural basis expansion analysis for interpretable time series forecasting
• Are All Languages Equally Hard to Language-Model?
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks
• Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
• Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

Cited by

• Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting

Related

• Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model
• TSPulse: Dual Space Tiny Pre-Trained Models for Rapid Time-Series Analysis
• Kairos: Towards Adaptive and Generalizable Time Series Foundation Models
• Rethinking Attention Mechanism in Time Series Classification
• Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series