2025/02/27 by Raphael Rossellini, Rossellini, Raphael, Jake A. Soloff +7 · 1 voice · 6 citations
Decision Sciences · Earth and Planetary Sciences · Mathematics · #FOS: Computer and information sciences #Forecasting Techniques and Applications #Machine Learning (stat.ML) #Meteorological Phenomena and Simulations #Methodology (stat.ME) #Risk and Portfolio Optimization #stat.ME #stat.ML
paper · pdf · doi:10.48550/arxiv.2502.19851
openalex publication_date 2025/02/27 · arxiv published 2025/02/27 · arxiv updated 2025/08/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Forecast probabilities often serve as critical inputs for binary decision making. In such settings, calibration\unicodex2014ensuring forecasted probabilities match empirical frequencies\unicodex2014is essential. Although the common notion of Expected Calibration Error (ECE) provides actionable insights for decision making, it is not testable: it cannot be empirically estimated in many practical cases. Conversely, the recently proposed Distance from Calibration (dCE) is testable, but it is not actionable since it lacks decision-theoretic guarantees needed for high-stakes applications. To resolve this question, we consider Cutoff Calibration Error, a calibration measure that bridges this gap by assessing calibration over intervals of forecasted probabilities. We show that Cutoff Calibration Error is both testable and actionable, and we examine its implications for popular post-hoc calibration methods, such as isotonic regression and Platt scaling.