2022/02/06 by Arjun Soin, Jameson Merkow, Soin, Arjun +15 · 1 citation
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence in Healthcare and Education #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #Data Stream Mining Techniques #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Machine Learning in Healthcare #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2202.02833
openalex publication_date 2022/02/06 · openalex created_date 2022/04/03 · openalex updated_date 2026/07/28
Clinical Artificial lntelligence (AI) applications are rapidly expanding worldwide, and have the potential to impact to all areas of medical practice. Medical imaging applications constitute a vast majority of approved clinical AI applications. Though healthcare systems are eager to adopt AI solutions a fundamental question remains: what happens after the AI model goes into production? We use the CheXpert and PadChest public datasets to build and test a medical imaging AI drift monitoring workflow to track data and model drift without contemporaneous ground truth. We simulate drift in multiple experiments to compare model performance with our novel multi-modal drift metric, which uses DICOM metadata, image appearance representation from a variational autoencoder (VAE), and model output probabilities as input. Through experimentation, we demonstrate a strong proxy for ground truth performance using unsupervised distributional shifts in relevant metadata, predicted probabilities, and VAE latent representation. Our key contributions include (1) proof-of-concept for medical imaging drift detection that includes the use of VAE and domain specific statistical methods, (2) a multi-modal methodology to measure and unify drift metrics, (3) new insights into the challenges and solutions to observe deployed medical imaging AI, and (4) creation of open-source tools that enable others to easily run their own workflows and scenarios. This work has important implications. It addresses the concerning translation gap found in continuous medical imaging AI model monitoring common in dynamic healthcare environments.