2023/11/24 by Vasantha Kumar Venugopal, Venugopal, Vasantha Kumar, Abhishek Gupta +5
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Radiomics and Machine Learning in Medical Imaging
paper · pdf · doi:10.48550/arxiv.2311.14305
openalex publication_date 2023/11/24 · openalex created_date 2023/11/28 · openalex updated_date 2026/07/28
With the increasingly widespread adoption of AI in healthcare, maintaining the accuracy and reliability of AI models in clinical practice has become crucial. In this context, we introduce novel methods for monitoring the performance of radiology AI classification models in practice, addressing the challenges of obtaining real-time ground truth for performance monitoring. We propose two metrics - predictive divergence and temporal stability - to be used for preemptive alerts of AI performance changes. Predictive divergence, measured using Kullback-Leibler and Jensen-Shannon divergences, evaluates model accuracy by comparing predictions with those of two supplementary models. Temporal stability is assessed through a comparison of current predictions against historical moving averages, identifying potential model decay or data drift. This approach was retrospectively validated using chest X-ray data from a single-center imaging clinic, demonstrating its effectiveness in maintaining AI model reliability. By providing continuous, real-time insights into model performance, our system ensures the safe and effective use of AI in clinical decision-making, paving the way for more robust AI integration in healthcare