2022/09/02 by Yihao Li, Mostafa El Habib Daho, Li, Yihao +21 · 1 citation
Computer Science · Medicine · #Computer Vision and Pattern Recognition (cs.CV) #Digital Imaging for Blood Diseases #FOS: Computer and information sciences #FOS: Electrical engineering #Glaucoma and retinal disorders #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Retinal Imaging and Analysis #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2209.00979
openalex publication_date 2022/09/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Multimodal information is frequently available in medical tasks. By combining information from multiple sources, clinicians are able to make more accurate judgments. In recent years, multiple imaging techniques have been used in clinical practice for retinal analysis: 2D fundus photographs, 3D optical coherence tomography (OCT) and 3D OCT angiography, etc. Our paper investigates three multimodal information fusion strategies based on deep learning to solve retinal analysis tasks: early fusion, intermediate fusion, and hierarchical fusion. The commonly used early and intermediate fusions are simple but do not fully exploit the complementary information between modalities. We developed a hierarchical fusion approach that focuses on combining features across multiple dimensions of the network, as well as exploring the correlation between modalities. These approaches were applied to glaucoma and diabetic retinopathy classification, using the public GAMMA dataset (fundus photographs and OCT) and a private dataset of PlexElite 9000 (Carl Zeis Meditec Inc.) OCT angiography acquisitions, respectively. Our hierarchical fusion method performed the best in both cases and paved the way for better clinical diagnosis.