2022/05/26 by Dimitris Spathis, Spathis, Dimitris, Stephanie L. Hyland +1 · 1 citation
Computer Science · Medicine · #Emergency and Acute Care Studies #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Sepsis Diagnosis and Treatment
paper · pdf · doi:10.48550/arxiv.2205.13398
openalex publication_date 2022/05/26 · openalex created_date 2022/06/13 · openalex updated_date 2026/07/28
Clinical machine learning models show a significant performance drop when tested in settings not seen during training. Domain generalisation models promise to alleviate this problem, however, there is still scepticism about whether they improve over traditional training. In this work, we take a principled approach to identifying Out of Distribution (OoD) environments, motivated by the problem of cross-hospital generalization in critical care. We propose model-based and heuristic approaches to identify OoD environments and systematically compare models with different levels of held-out information. We find that access to OoD data does not translate to increased performance, pointing to inherent limitations in defining potential OoD environments potentially due to data harmonisation and sampling. Echoing similar results with other popular clinical benchmarks in the literature, new approaches are required to evaluate robust models on health records.