2017/02/10 by Jens Witkowski, Pavel Atanasov, Lyle Ungar +1 · 1 voice · 2 citations
Decision Sciences · Economics, Econometrics and Finance · #Decision-Making and Behavioral Economics #Forecasting Techniques and Applications #Sports Analytics and Performance
paper · doi:10.1609/aaai.v31i1.10590
openalex publication_date 2017/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.