Foundation vs. Specialized Models: Evaluating Catastrophic Forgetting in Continual Time Series Forecasting

2025/10/01 by Nouha Karaouli, Denis Coquenet, Karaouli, Nouha +7
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2510.00809

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While Time Series Foundation Models (TSFMs) excel in zero-shot tasks, their behavior under continual fine tuning is poorly understood. We present the first systematic study of catastrophic forgetting in TSFMs (TimesFM-2.0, Chronos-2) versus a specialized SamFormer model across synthetic and real-world energy forecasting benchmarks. Our results show that while fine-tuning improves new task accuracy, it consistently triggers forgetting, though larger models exhibit greater inherent robustness. Notably, employing forgetting mitigation techniques such as DER, levels the playing field: it provides disproportionate gains to smaller models, allowing them to match TSFM performance by the end of the continual learning sequence. These findings suggest that in realistic, non-stationary scenarios, the high computational cost of large foundation models may not be justified over smaller models equipped with effective mitigation strategies.

Citations

• Chronos-2: From Univariate to Universal Forecasting
• How Foundational are Foundation Models for Time Series Forecasting?
• TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
• Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuning
• GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation
• Continual Learning of Large Language Models: A Comprehensive Survey
• Chronos: Learning the Language of Time Series
• SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention
• Unified Training of Universal Time Series Forecasting Transformers
• A decoder-only foundation model for time-series forecasting
• Large Language Models Are Zero-Shot Time Series Forecasters
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
• A Comprehensive Survey of Continual Learning: Theory, Method and Application
• Continual Learning for Human State Monitoring
• Are Transformers Effective for Time Series Forecasting?
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Learning Transferable Visual Models From Natural Language Supervision
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Random sampling with a reservoir
• FlowState: Sampling Rate Invariant Time Series Forecasting
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuning
• How Foundational are Foundation Models for Time Series Forecasting?
• Exploring Representations and Interventions in Time Series Foundation Models
• Foundation Models and Fine-Tuning: Toward a New Generation of Models for Time Series Forecasting
• Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning