2025/09/17 by Menelaos Pavlou, Pavlou, Menelaos, Rumana Omar +3
Computer Science · Medicine · #Applications (stat.AP) #Artificial Intelligence in Healthcare and Education #Computation (stat.CO) #FOS: Computer and information sciences #Machine Learning in Healthcare #Methodology (stat.ME) #Sepsis Diagnosis and Treatment
paper · pdf · doi:10.48550/arxiv.2509.14028
openalex publication_date 2025/09/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Sample size calculations for developing prediction models typically aim to ensure that the expected value of a performance measure meets a prespecified target. For example, a key measure is the calibration slope (CS) which quantifies model overfitting; the sample size is often chosen so that the expected CS equals 0.9, close to the ideal value of 1. However, because of sampling variability, model performance can vary substantially across development samples of the recommended size. When variability is high, the probability of obtaining a model with performance close to the target may be unacceptably low. We propose a framework which enables sample size calculations that incorporate both the expected value and the variability for a given performance measure. The framework is illustrated for binary outcomes and logistic regression but applies to other outcomes and model types. To explicitly account for variability, we introduce the probability of acceptable performance (PrAP). For example, a model may be considered acceptable if the CS lies within a prespecified range (e.g., 0.85 to 1.15) and the sample size might be chosen to ensure that PrAP exceeds some target (e.g., 80%). Under existing approaches PrAP can be low, especially when the specified number of predictors is small, which can also translate into large variability for individual predicted probabilities. The use of shrinkage tends to improve PrAP. Our findings highlight the importance of accounting for performance variability to ensure robust model development.