2023/04/19 by Étienne Marcotte, Marcotte, Étienne, Valentina Zantedeschi +5 · 1 citation
Decision Sciences · Engineering · #Energy Load and Power Forecasting #FOS: Computer and information sciences #Forecasting Techniques and Applications #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stock Market Forecasting Methods
paper · pdf · doi:10.48550/arxiv.2304.09836
openalex publication_date 2023/04/19 · openalex created_date 2023/04/22 · openalex updated_date 2026/07/28
Multivariate probabilistic time series forecasts are commonly evaluated via proper scoring rules, i.e., functions that are minimal in expectation for the ground-truth distribution. However, this property is not sufficient to guarantee good discrimination in the non-asymptotic regime. In this paper, we provide the first systematic finite-sample study of proper scoring rules for time-series forecasting evaluation. Through a power analysis, we identify the "region of reliability" of a scoring rule, i.e., the set of practical conditions where it can be relied on to identify forecasting errors. We carry out our analysis on a comprehensive synthetic benchmark, specifically designed to test several key discrepancies between ground-truth and forecast distributions, and we gauge the generalizability of our findings to real-world tasks with an application to an electricity production problem. Our results reveal critical shortcomings in the evaluation of multivariate probabilistic forecasts as commonly performed in the literature.