2023/07/05 by James E. Jackson, Jackson, James, Robin Mitra +7 · 2 citations
Computer Science · Mathematics · #Applications (stat.AP) #Bayesian Modeling and Causal Inference #Data Mining Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (stat.ML) #Methodology (stat.ME) #Statistical Methods and Bayesian Inference
paper · pdf · doi:10.48550/arxiv.2307.02650
openalex publication_date 2023/07/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Our capacity to process large complex data sources is ever-increasing, providing us with new, important applied research questions to address, such as how to handle missing values in large-scale databases. Mitra et al. (2023) noted the phenomenon of Structured Missingness (SM), which is where missingness has an underlying structure. Existing taxonomies for defining missingness mechanisms typically assume that variables' missingness indicator vectors M1, M2, ..., Mp are independent after conditioning on the relevant portion of the data matrix X. As this is often unsuitable for characterising SM in multivariate settings, we introduce a taxonomy for SM, where each Mj can depend on M-j (i.e., all missingness indicator vectors except Mj), in addition to X. We embed this new framework within the well-established decomposition of mechanisms into MCAR, MAR, and MNAR (Rubin, 1976), allowing us to recast mechanisms into a broader setting, where we can consider the combined effect of X and M-j on Mj. We also demonstrate, via simulations, the impact of SM on inference and prediction, and consider contextual instances of SM arising in a de-identified nationwide (US-based) clinico-genomic database (CGDB). We hope to stimulate interest in SM, and encourage timely research into this phenomenon.