2020/12/06 by Nishant Kumar, Kumar, Nishant, Stefan Gumhold +1
Computer Science · Engineering · Neuroscience · #Advanced Image Fusion Techniques #Brain Tumor Detection and Classification #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Visual Attention and Saliency Detection #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2012.08932
openalex publication_date 2020/12/06 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Image fusion helps in merging two or more images to construct a more\ninformative single fused image. Recently, unsupervised learning based\nconvolutional neural networks (CNN) have been utilized for different types of\nimage fusion tasks such as medical image fusion, infrared-visible image fusion\nfor autonomous driving as well as multi-focus and multi-exposure image fusion\nfor satellite imagery. However, it is challenging to analyze the reliability of\nthese CNNs for the image fusion tasks since no groundtruth is available. This\nled to the use of a wide variety of model architectures and optimization\nfunctions yielding quite different fusion results. Additionally, due to the\nhighly opaque nature of such neural networks, it is difficult to explain the\ninternal mechanics behind its fusion results. To overcome these challenges, we\npresent a novel real-time visualization tool, named FuseVis, with which the\nend-user can compute per-pixel saliency maps that examine the influence of the\ninput image pixels on each pixel of the fused image. We trained several image\nfusion based CNNs on medical image pairs and then using our FuseVis tool, we\nperformed case studies on a specific clinical application by interpreting the\nsaliency maps from each of the fusion methods. We specifically visualized the\nrelative influence of each input image on the predictions of the fused image\nand showed that some of the evaluated image fusion methods are better suited\nfor the specific clinical application. To the best of our knowledge, currently,\nthere is no approach for visual analysis of neural networks for image fusion.\nTherefore, this work opens up a new research direction to improve the\ninterpretability of deep fusion networks. The FuseVis tool can also be adapted\nin other deep neural network based image processing applications to make them\ninterpretable.\n