Online Conformal Model Selection for Nonstationary Time Series

2025/06/05 by Shibo Li, Yao Zheng, Li, Shibo +1
Computer Science · Mathematics · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistical Methods and Inference #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2506.05544

openalex publication_date 2025/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces the MPS (Model Prediction Set), a novel framework for online model selection for nonstationary time series. Classical model selection methods, such as information criteria and cross-validation, rely heavily on the stationarity assumption and often fail in dynamic environments which undergo gradual or abrupt changes over time. Yet real-world data are rarely stationary, and model selection under nonstationarity remains a largely open problem. To tackle this challenge, we combine conformal inference with model confidence sets to develop a procedure that adaptively selects models best suited to the evolving dynamics at any given time. Concretely, the MPS updates in real time a confidence set of candidate models that covers the best model for the next time period with a specified long-run probability, while adapting to nonstationarity of unknown forms. Through simulations and real-world data analysis, we demonstrate that MPS reliably and efficiently identifies optimal models under nonstationarity, an essential capability lacking in offline methods. Moreover, MPS frequently produces high-quality sets with small cardinality, whose evolution offers deeper insights into changing dynamics. As a generic framework, MPS accommodates any data-generating process, data structure, model class, training method, and evaluation metric, making it broadly applicable across diverse problem settings.

Citations

• Model Class Selection
• Conditional Method Confidence Set
• Conformal inference for random objects
• Sequential model confidence sets
• Bellman Conformal Inference: Calibrating Prediction Intervals For Time Series
• Efficient Methods for Non-stationary Online Learning
• Non-stationary Projection-free Online Learning with Dynamic and Adaptive Regret Guarantees
• Cox reduction and confidence sets of models: a theoretical elucidation
• Improved Online Conformal Prediction via Strongly Adaptive Online Learning
• Finite- and Large- Sample Inference for Model and Coefficients in High-dimensional Linear Regression with Repro Samples
• Conformal Inference for Online Prediction with Arbitrary Distribution Shifts
• Adaptive Conformal Predictions for Time Series
• Adaptive Conformal Inference Under Distribution Shift
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Structural Breaks in Time Series
• Model Selection Confidence Sets by Likelihood Ratio Testing
• Attention Is All You Need
• Model Confidence Bounds for Variable Selection
• Predictive, finite-sample model choice for time series under stationarity and non-stationarity
• Distribution-Free Predictive Inference For Regression
• Locally Stationary Processes
• Specification, Estimation, and Evaluation of Smooth Transition Autoregressive Models
• Reinforcement Learning: An Introduction

Related

• Conformal Prediction for Time-series Forecasting with Change Points
• Adaptive Conformal Predictions for Time Series
• Foundation models for time series forecasting: Application in conformal prediction
• Conformal PID Control for Time Series Prediction
• Online conformal inference for multi-step time series forecasting