3D-StyleGAN: A Style-Based Generative Adversarial Network for Generative Modeling of Three-Dimensional Medical Images

2021/07/20 by Sung‐Min Hong, Hong, Sungmin, Razvan Marinescu +11 · 3 citations
Computer Science · #68T07 (Primary) 68T01 (Secondary) #Advanced Vision and Imaging #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Generative Adversarial Networks and Image Synthesis #I.2 #I.4 #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2107.09700

openalex publication_date 2021/07/20 · openalex created_date 2021/08/02 · openalex updated_date 2026/07/28

Abstract

Image synthesis via Generative Adversarial Networks (GANs) of three-dimensional (3D) medical images has great potential that can be extended to many medical applications, such as, image enhancement and disease progression modeling. However, current GAN technologies for 3D medical image synthesis need to be significantly improved to be readily adapted to real-world medical problems. In this paper, we extend the state-of-the-art StyleGAN2 model, which natively works with two-dimensional images, to enable 3D image synthesis. In addition to the image synthesis, we investigate the controllability and interpretability of the 3D-StyleGAN via style vectors inherited form the original StyleGAN2 that are highly suitable for medical applications: (i) the latent space projection and reconstruction of unseen real images, and (ii) style mixing. We demonstrate the 3D-StyleGAN's performance and feasibility with ~12,000 three-dimensional full brain MR T1 images, although it can be applied to any 3D volumetric images. Furthermore, we explore different configurations of hyperparameters to investigate potential improvement of the image synthesis with larger networks. The codes and pre-trained networks are available online: https://github.com/sh4174/3DStyleGAN.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Generative Adversarial Networks
• Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Training Generative Adversarial Networks with Limited Data
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Generative Adversarial Networks

Cited by

• USB: Unified Synthetic Brain Framework for Bidirectional Pathology-Healthy Generation and Editing
• Semantics versus Identity: A Divide-and-Conquer Approach towards Adjustable Medical Image De-Identification
• TumorGen: Boundary-Aware Tumor-Mask Synthesis with Rectified Flow Matching

Related

• GANs for Medical Image Synthesis: An Empirical Study
• Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data
• Medical Image Generation using Generative Adversarial Networks
• Deep Generative Models for 3D Medical Image Synthesis
• Three-dimensional Bone Image Synthesis with Generative Adversarial Networks