GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation

2024/10/14 by Taha Aksu, Gerald Woo, Aksu, Taha +13 · 45 citations
Computer Science · Decision Sciences · #FOS: Computer and information sciences #Forecasting Techniques and Applications #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stock Market Forecasting Methods #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2410.10393

openalex publication_date 2024/10/14 · openalex created_date 2024/10/20 · openalex updated_date 2026/07/28

Abstract

Time series foundation models excel in zero-shot forecasting, handling diverse tasks without explicit training. However, the advancement of these models has been hindered by the lack of comprehensive benchmarks. To address this gap, we introduce the General Time Series Forecasting Model Evaluation, GIFT-Eval, a pioneering benchmark aimed at promoting evaluation across diverse datasets. GIFT-Eval encompasses 23 datasets over 144,000 time series and 177 million data points, spanning seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts. To facilitate the effective pretraining and evaluation of foundation models, we also provide a non-leaking pretraining dataset containing approximately 230 billion data points. Additionally, we provide a comprehensive analysis of 17 baselines, which includes statistical models, deep learning models, and foundation models. We discuss each model in the context of various benchmark characteristics and offer a qualitative analysis that spans both deep learning and foundation models. We believe the insights from this analysis, along with access to this new standard zero-shot time series forecasting benchmark, will guide future developments in time series foundation models. Code, data, and the leaderboard can be found at https://github.com/SalesforceAIResearch/gift-eval .

Cited by

• Are Time-Indexed Foundation Models the Future of Time Series Imputation?
• Time Series Foundation Models for Process Model Forecasting
• Foundation Models and Fine-Tuning: Toward a New Generation of Models for Time Series Forecasting
• Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting
• TS-Arena Technical Report -- A Pre-registered Live Forecasting Platform
• Modèles de Fondation et Ajustement : Vers une Nouvelle Génération de Modèles pour la Prévision des Séries Temporelles
• Tiny-TSM: Efficiently Training a Lightweight SOTA Time Series Foundation Model
• Cisco Time Series Model Technical Report
• Scaling Conditional Autoencoders for Portfolio Optimization via Uncertainty-Aware Factor Selection
• Multi-layer Stack Ensembles for Time Series Forecasting
• Moirai 2.0: When Less Is More for Time Series Forecasting
• Synapse: Adaptive Arbitration of Complementary Expertise in Time Series Foundational Models
• Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification
• TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting
• Chronos-2: From Univariate to Universal Forecasting
• SEMPO: Lightweight Foundation Models for Time Series Forecasting
• Benchmarking Probabilistic Time Series Forecasting Models on Neural Activity
• Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention
• Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?
• Tackling Time-Series Forecasting Generalization via Mitigating Concept Drift
• Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges
• KAIROS: Unified Training for Universal Non-Autoregressive Time Series Forecasting
• Test-Time Efficient Pretrained Model Portfolios for Time Series Forecasting
• TelecomTS: A Multi-Modal Observability Dataset for Time Series and Language Analysis
• Benchmarking M-LTSF: Frequency and Noise-Based Evaluation of Multivariate Long Time Series Forecasting Models
• OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data
• Accuracy Law for the Future of Deep Time Series Forecasting
• TsLLM: Augmenting LLMs for General Time Series Understanding and Prediction
• Foundation vs. Specialized Models: Evaluating Catastrophic Forgetting in Continual Time Series Forecasting
• How Foundational are Foundation Models for Time Series Forecasting?
• fev-bench: A Realistic Benchmark for Time Series Forecasting
• Kairos: Towards Adaptive and Generalizable Time Series Foundation Models
• Estimating Time Series Foundation Model Transferability via In-Context Learning
• How Effective Are Time-Series Models for Precipitation Nowcasting? A Comprehensive Benchmark for GNSS-based Precipitation Nowcasting
• From Samples to Scenarios: A New Paradigm for Probabilistic Forecasting
• A More Realistic Evaluation of Cross-Frequency Transfer Learning and Foundation Forecasting Models
• Tsururu: A Python-based Time Series Forecasting Strategies Library
• Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecasting
• TFMAdapter: Lightweight Instance-Level Adaptation of Foundation Models for Forecasting with Covariates
• ARIES: Relation Assessment and Model Recommendation for Deep Time Series Forecasting
• One-Embedding-Fits-All: Efficient Zero-Shot Time Series Forecasting by a Model Zoo
• When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference
• TimeCopilot
• VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbones
• CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data

Related

• TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting
• Foundation Models and Fine-Tuning: Toward a New Generation of Models for Time Series Forecasting
• Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges
• Conversational Time Series Foundation Models: Towards Explainable and Effective Forecasting
• One-Embedding-Fits-All: Efficient Zero-Shot Time Series Forecasting by a Model Zoo