2022/11/26 by Zixiang Zhao, Zhao, Zixiang, Haowen Bai +13 · 67 citations
Engineering · #Advanced Image Fusion Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Photoacoustic and Ultrasonic Imaging #Remote-Sensing Image Classification
paper · pdf · doi:10.48550/arxiv.2211.14461
openalex publication_date 2022/11/26 · openalex created_date 2022/12/10 · openalex updated_date 2026/08/01
Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.com/Zhaozixiang1228/MMIF-CDDFuse.