2026/02/13 by Rajveer Singh Pall, Sameer Yadav, Siddharth Bhalerao +3
Health Professions · Computer Science · Medicine · #Artificial Intelligence in Healthcare #Machine Learning in Healthcare #Artificial Intelligence in Healthcare and Education
paper · pdf · doi:10.1109/cipher70417.2026.11523789
Machine learning-based Type 2 diabetes risk prediction models mostly obtain good internal validation results but lose their effectiveness in real-world applications due to deficient external testing and fairness assessment. We developed a comprehensive, multi-dimensional framework for evaluating discrimination, calibration, interpretability, and algorithmic fairness when applied to nationally representative populations. An XGBoost model and baseline models were trained on NHANES 2015-2020 (n = 15,685) using eight non-laboratory predictors: age, sex, race/ethnicity, BMI, smoking status, physical activity, history of heart attack, and history of stroke. External validation was performed with BRFSS 2020–2022 (n=1,285,783), a phone-based surveillance system that has realistic deployment conditions and a large distribution shift. Internal validation showed good discrimination (AUC 0.794, 95% CI 0.788 to 0.800), but there was some loss of performance on external validation (AUC=0.717, relative decrease: −9.7%, p < 0.001). Fairness analysis revealed severe bias: predictions for elderly adults (≥60) were much worse than for young adults (AUC 0.607 vs 0.742, difference 0.135, p < 0.001), the obese underperformed compared to normal weight individuals (AUC 0.698 vs 0.735, difference of 0.037). Gender demonstrated comparability (male 0.723 versus female 0.712, p =0.142). Calibration showed that there was an overestimation of the risk (Brier score 0.123). The SHAP analysis showed that age, BMI, and physical activity were the main risk drivers with a clinically consistent direction. Internal validation severely overestimates real-life performance. Massive external validation revealed systematic errors in the most vulnerable: those who are old and obese, who have the highest risk of diabetes, get a worse AI-based score. Such differences call for the development of equity consideration-based and age-specific deployment approaches before any clinical use.