2024/12/13 by Ben Van Calster, Van Calster, Ben, Professor Gary S. Collins +32 · 7 voices · 14 citations
Computer Science · Economics, Econometrics and Finance · Health Professions · Mathematics · Medicine · #Artificial Intelligence in Healthcare and Education #Artificial intelligence #Calibration #Computer science #Data mining #Econometrics #Health Systems, Economic Evaluations, Quality of Life #Healthcare cost, quality, practices #Machine learning #Mathematics #Measure (data warehouse) #Plot (graphics) #Predictive modelling #Receiver operating characteristic #Statistics #cs.LG #stat.ME #stat.ML
paper · pdf · doi:10.48550/arxiv.2412.10288
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/12/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
A myriad of measures to illustrate performance of predictive artificial\nintelligence (AI) models have been proposed in the literature. Selecting\nappropriate performance measures is essential for predictive AI models that are\ndeveloped to be used in medical practice, because poorly performing models may\nharm patients and lead to increased costs. We aim to assess the merits of\nclassic and contemporary performance measures when validating predictive AI\nmodels for use in medical practice. We focus on models with a binary outcome.\nWe discuss 32 performance measures covering five performance domains\n(discrimination, calibration, overall, classification, and clinical utility)\nalong with accompanying graphical assessments. The first four domains cover\nstatistical performance, the fifth domain covers decision-analytic performance.\nWe explain why two key characteristics are important when selecting which\nperformance measures to assess: (1) whether the measure's expected value is\noptimized when it is calculated using the correct probabilities (i.e., a\n"proper" measure), and (2) whether they reflect either purely statistical\nperformance or decision-analytic performance by properly considering\nmisclassification costs. Seventeen measures exhibit both characteristics,\nfourteen measures exhibited one characteristic, and one measure possessed\nneither characteristic (the F1 measure). All classification measures (such as\nclassification accuracy and F1) are improper for clinically relevant decision\nthresholds other than 0.5 or the prevalence. We recommend the following\nmeasures and plots as essential to report: AUROC, calibration plot, a clinical\nutility measure such as net benefit with decision curve analysis, and a plot\nwith probability distributions per outcome category.\n