Hidden Stratification Causes Clinically Meaningful Failures in Machine\n Learning for Medical Imaging

2019/09/26 by Luke Oakden‐Rayner, Jared Dunnmon, Oakden-Rayner, Luke +5 · 18 citations
Medicine · Computer Science · Mathematics · #Radiomics and Machine Learning in Medical Imaging #AI in cancer detection #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.1909.12475

Abstract

Machine learning models for medical image analysis often suffer from poor\nperformance on important subsets of a population that are not identified during\ntraining or testing. For example, overall performance of a cancer detection\nmodel may be high, but the model still consistently misses a rare but\naggressive cancer subtype. We refer to this problem as hidden stratification,\nand observe that it results from incompletely describing the meaningful\nvariation in a dataset. While hidden stratification can substantially reduce\nthe clinical efficacy of machine learning models, its effects remain difficult\nto measure. In this work, we assess the utility of several possible techniques\nfor measuring and describing hidden stratification effects, and characterize\nthese effects on multiple medical imaging datasets. We find evidence that\nhidden stratification can occur in unidentified imaging subsets with low\nprevalence, low label quality, subtle distinguishing features, or spurious\ncorrelates, and that it can result in relative performance differences of over\n20% on clinically important subsets. Finally, we explore the clinical\nimplications of our findings, and suggest that evaluation of hidden\nstratification should be a critical component of any machine learning\ndeployment in medical imaging.\n

Cited by

• Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples
• Localising Shortcut Learning in Pixel Space via Ordinal Scoring Correlations for Attribution Representations (OSCAR)
• meval: A Statistical Toolbox for Fine-Grained Model Performance Analysis
• TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models
• Accounting for Underspecification in Statistical Claims of Model Superiority
• Gaps in artificial intelligence research for rural health in the United States: a scoping review
• FT-ARM: Fine-Tuned Agentic Reflection Multimodal Language Model for Pressure Ulcer Severity Classification with Reasoning
• Adoption paradox of artificial intelligence in computational pathology: a three-stage maturity model from algorithms to clinical integration
• Outlier Detection in Plantar Pressure: Human-Centered Comparison of Statistical Parametric Mapping and Explainable Machine Learning
• Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context
• From Lesion to Decision: AI for ARIA Detection and Predictive Imaging in Alzheimer’s Disease
• Probabilistic Runtime Verification, Evaluation and Risk Assessment of Visual Deep Learning Systems
• Invisible Attributes, Visible Biases: Exploring Demographic Shortcuts in MRI-based Alzheimer's Disease Classification
• Detecting Regional Spurious Correlations in Vision Transformers via Token Discarding
• Advanced Multi-Architecture Deep Learning Framework for BIRADS-Based Mammographic Image Retrieval: Comprehensive Performance Analysis with Super-Ensemble Optimization
• Label-free estimation of clinically relevant performance metrics under distribution shifts
• Predict Patient Self-reported Race from Skin Histological Images
• Fairness and Robustness of CLIP-Based Models for Chest X-rays

Related

• Accounting for Underspecification in Statistical Claims of Model Superiority
• Does Medical Imaging learn different Convolution Filters?
• Learning to Unlearn: Building Immunity to Dataset Bias in Medical Imaging Studies
• When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations
• Algorithmic encoding of protected characteristics in image-based models for disease detection