2018/06/10 by Julie Bessac, Philippe Naveau, Bessac, Julie +1
Decision Sciences · Earth and Planetary Sciences · Environmental Science · #Atmospheric and Environmental Gas Dynamics #FOS: Computer and information sciences #Forecasting Techniques and Applications #Meteorological Phenomena and Simulations #Methodology (stat.ME)
paper · pdf · doi:10.48550/arxiv.1806.03745
openalex publication_date 2018/06/10 · openalex created_date 2018/06/13 · openalex updated_date 2026/07/28
The classical paradigm of scoring rules is to discriminate between two\ndifferent forecasts by comparing them with observations. The probability\ndistribution of the observed record is assumed to be perfect as a verification\nbenchmark. In practice, however, observations are almost always tainted by\nerrors and uncertainties. If the yardstick used to compare forecasts is\nimprecise, one can wonder whether such types of errors may or may not have a\nstrong influence on decisions based on classical scoring rules. We propose a\nnew scoring rule scheme in the context of models that incorporate errors of the\nverification data. We rely on existing scoring rules and incorporate\nuncertainty and error of the verification data through a hidden variable and\nthe conditional expectation of scores when they are viewed as a random\nvariable. The proposed scoring framework is compared to scores used in\npractice, and is expressed in various setups, mainly an additive Gaussian noise\nmodel and a multiplicative Gamma noise model. By considering scores as random\nvariables one can access the entire range of their distribution. In particular\nwe illustrate that the commonly used mean score can be a misleading\nrepresentative of the distribution when this latter is highly skewed or have\nheavy tails. In a simulation study, through the power of a statistical test and\nthe computation of Wasserstein distances between scores distributions, we\ndemonstrate the ability of the newly proposed score to better discriminate\nbetween forecasts when verification data are subject to uncertainty compared\nwith the scores used in practice. Finally, we illustrate the benefit of\naccounting for the uncertainty of the verification data into the scoring\nprocedure on a dataset of surface wind speed from measurements and numerical\nmodel outputs.\n