Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

2015/11/09 by Gary S. Collins, Professor Gary S. Collins, Emmanuel O. Ogundimu +2 · 11 citations
Mathematics · #Statistical Methods in Epidemiology #Statistical Methods and Bayesian Inference #Statistical Methods and Inference

paper · pdf · doi:10.1002/sim.6787

Abstract

After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c-index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events.

Citations

• Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration
• Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement
• Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement

Cited by

• Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
• Updated 5-year and new 10-year sexual recidivism rate norms for Static-99R with routine/complete samples.
• Sample size for binary logistic prediction models: Beyond events per variable criteria
• There is no such thing as a validated prediction model
• The importance of being external. methodological insights for the external validation of machine learning models in medicine
• Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal
• External validation of prognostic models: what, why, how, when and where?
• Depression diagnosis from patient interviews using multimodal machine learning
• Using Natural Language Processing on Electronic Health Records to Enhance Detection and Prediction of Psychosis Risk
• Association between periodontal diseases and cardiovascular diseases, diabetes and respiratory diseases: Consensus report of the Joint Workshop by the European Federation of Periodontology (EFP) and the European arm of the World Organization of Family Doctors (WONCA Europe)
• Calculating the sample size required for developing a clinical prediction model

Related

• External validation of prognostic models: what, why, how, when and where?
• Minimum sample size for developing a multivariable prediction model: PART II ‐ binary and time‐to‐event outcomes
• Bayesian sample size calculations for external validation studies of risk prediction models
• Formalising Sample Size Calculations for the Development of Risk Prediction Models: The Importance of Accounting for Performance Variability
• Extended sample size calculations for evaluation of prediction models using a threshold for classification