2018/11/18 by Faisal Mahmood, Ziyun Yang, Mahmood, Faisal +5 · 1 citation
Computer Science · Engineering · Medicine · #AI in cancer detection #Advanced Image Fusion Techniques #Artificial Intelligence (cs.AI) #Artificial intelligence #Colorectal Cancer Screening and Detection #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Concatenation (mathematics) #Deep learning #FOS: Computer and information sciences #Flexibility (engineering) #Fuse (electrical) #Identification (biology) #Machine Learning (cs.LG) #Machine learning #Margin (machine learning) #Modalities #Pattern recognition (psychology) #cs.AI #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.1811.07407
10 pages, 6 figures
arxiv created 2018/11/18 · openalex publication_date 2018/11/18 · arxiv updated 2018/11/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Humans make accurate decisions by interpreting complex data from multiple sources. Medical diagnostics, in particular, often hinge on human interpretation of multi-modal information. In order for artificial intelligence to make progress in automated, objective, and accurate diagnosis and prognosis, methods to fuse information from multiple medical imaging modalities are required. However, combining information from multiple data sources has several challenges, as current deep learning architectures lack the ability to extract useful representations from multimodal information, and often simple concatenation is used to fuse such information. In this work, we propose Multimodal DenseNet, a novel architecture for fusing multimodal data. Instead of focusing on concatenation or early and late fusion, our proposed architectures fuses information over several layers and gives the model flexibility in how it combines information from multiple sources. We apply this architecture to the challenge of polyp characterization and landmark identification in endoscopy. Features from white light images are fused with features from narrow band imaging or depth maps. This study demonstrates that Multimodal DenseNet outperforms monomodal classification as well as other multimodal fusion techniques by a significant margin on two different datasets.