2022/12/18 by Firas Khader, Khader, Firas, Gustav Mueller-Franzes +23
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #COVID-19 diagnosis using AI #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Retinal Imaging and Analysis #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2212.09162
openalex publication_date 2022/12/18 · arxiv created 2022/12/20 · arxiv updated 2022/12/21 · openalex created_date 2023/01/04 · openalex updated_date 2026/07/28
Multimodal deep learning has been used to predict clinical endpoints and diagnoses from clinical routine data. However, these models suffer from scaling issues: they have to learn pairwise interactions between each piece of information in each data type, thereby escalating model complexity beyond manageable scales. This has so far precluded a widespread use of multimodal deep learning. Here, we present a new technical approach of "learnable synergies", in which the model only selects relevant interactions between data modalities and keeps an "internal memory" of relevant data. Our approach is easily scalable and naturally adapts to multimodal data inputs from clinical routine. We demonstrate this approach on three large multimodal datasets from radiology and ophthalmology and show that it outperforms state-of-the-art models in clinically relevant diagnosis tasks. Our new approach is transferable and will allow the application of multimodal deep learning to a broad set of clinically relevant problems.