Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative\n Adversarial Network

2019/04/27 by Anmol Sharma, Sharma, Anmol, Ghassan Hamarneh +1 · 11 citations
Computer Science · #Image and Signal Denoising Methods #Generative Adversarial Networks and Image Synthesis #Advanced Image Processing Techniques

paper · pdf · doi:10.48550/arxiv.1904.12200

Abstract

Magnetic resonance imaging (MRI) is being increasingly utilized to assess,\ndiagnose, and plan treatment for a variety of diseases. The ability to\nvisualize tissue in varied contrasts in the form of MR pulse sequences in a\nsingle scan provides valuable insights to physicians, as well as enabling\nautomated systems performing downstream analysis. However many issues like\nprohibitive scan time, image corruption, different acquisition protocols, or\nallergies to certain contrast materials may hinder the process of acquiring\nmultiple sequences for a patient. This poses challenges to both physicians and\nautomated systems since complementary information provided by the missing\nsequences is lost. In this paper, we propose a variant of generative\nadversarial network (GAN) capable of leveraging redundant information contained\nwithin multiple available sequences in order to generate one or more missing\nsequences for a patient scan. The proposed network is designed as a\nmulti-input, multi-output network which combines information from all the\navailable pulse sequences, implicitly infers which sequences are missing, and\nsynthesizes the missing ones in a single forward pass. We demonstrate and\nvalidate our method on two brain MRI datasets each with four sequences, and\nshow the applicability of the proposed method in simultaneously synthesizing\nall missing sequences in any possible scenario where either one, two, or three\nof the four sequences may be missing. We compare our approach with competing\nunimodal and multi-modal methods, and show that we outperform both\nquantitatively and qualitatively.\n

Cited by

• Deep infant brain segmentation from multi-contrast MRI
• TARDis: Time Attenuated Representation Disentanglement for Incomplete Multi-Modal Tumor Segmentation and Classification
• CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
• DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities
• Predicting before Reconstruction: A generative prior framework for MRI acceleration
• CEPerFed: Communication-Efficient Personalized Federated Learning for Multi-Pulse MRI Classification
• Med-K2N: Flexible K-to-N Modality Translation for Medical Image Synthesis
• Clinical Multi-modal Fusion with Heterogeneous Graph and Disease Correlation Learning for Multi-Disease Prediction
• Integrating Anatomical Priors into a Causal Diffusion Model
• CHARM: Collaborative Harmonization across Arbitrary Modalities for Modality-agnostic Semantic Segmentation
• Pyramid Hierarchical Masked Diffusion Model for Imaging Synthesis

Related

• DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis
• Synergistic Reconstruction and Synthesis via Generative Adversarial\n Networks for Accelerated Multi-Contrast MRI
• Synthesizing Missing MRI Sequences from Available Modalities using Generative Adversarial Networks in BraTS Dataset
• Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks
• Multi-modal Contrastive Learning for Tumor-specific Missing Modality Synthesis