Confidence Intervals for Random Forest Permutation Importance with Missing Data

2025/07/18 by Nico Föge, Markus Pauly, Föge, Nico +1
Computer Science · Mathematics · #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning in Healthcare #Methodology (stat.ME) #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2507.13918

openalex publication_date 2025/07/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Random Forests are renowned for their predictive accuracy, but valid inference, particularly about permutation-based feature importances, remains challenging. Existing methods, such as the confidence intervals (CIs) from Ishwaran et al. (2019), are promising but assume complete feature observation. However, real-world data often contains missing values. In this paper, we investigate how common imputation techniques affect the validity of Random Forest permutation-importance CIs when data are incomplete. Through an extensive simulation and real-world benchmark study, we compare state-of-the-art imputation methods across various missing-data mechanisms and missing rates. Our results show that single-imputation strategies lead to low CI coverage. As a remedy, we adapt Rubin's rule to aggregate feature-importance estimates and their variances over several imputed datasets and account for imputation uncertainty. Our numerical results indicate that the adjusted CIs achieve better nominal coverage.

Citations

• Which Imputation Fits Which Feature Selection Method? A Survey-Based Simulation Study
• A Central Limit Theorem for the permutation importance measure
• Pure interaction effects unseen by Random Forests
• Evaluating tree-based imputation methods as an alternative to MICE PMM for drawing inference in empirical studies
• Consistency of Random Forest Type Algorithms under a Probabilistic Impurity Decrease Condition
• Multiple Imputation Through XGBoost
• A Note on Listwise Deletion versus Multiple Imputation
• A cautionary tale on using imputation methods for inference in matched pairs design
• Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife
• Correlation and variable importance in random forests
• mice: Multivariate Imputation by Chained Equations inR
• Multiple imputation using chained equations: Issues and guidance for practice
• Missing-Data Adjustments in Large Surveys

Related

• Random Forest Missing Data Algorithms
• Asymptotic Unbiasedness of the Permutation Importance Measure in Random Forest Models
• TRIP: A Nonparametric Test to Diagnose Biased Feature Importance Scores
• A Central Limit Theorem for the permutation importance measure
• A Debiased MDI Feature Importance Measure for Random Forests