Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

2021/04/12 by Daiqing Li, Li, Daiqing, Junlin Yang +7 · 8 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2104.05833

openalex publication_date 2021/04/12 · openalex created_date 2021/04/26 · openalex updated_date 2026/07/28

Abstract

Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human annotation efforts. This is the goal of semi-supervised learning, which exploits more widely available unlabeled data to complement small labeled data sets. In this paper, we propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. Concretely, we learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images supplemented with only few labeled ones. We build our architecture on top of StyleGAN2, augmented with a label synthesis branch. Image labeling at test time is achieved by first embedding the target image into the joint latent space via an encoder network and test-time optimization, and then generating the label from the inferred embedding. We evaluate our approach in two important domains: medical image segmentation and part-based face segmentation. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization, such as transferring from CT to MRI in medical imaging, and photographs of real faces to paintings, sculptures, and even cartoons and animal faces. Project Page: \urlhttps://nv-tlabs.github.io/semanticGAN/

Citations

• Adversarial Feature Learning
• Precise Recovery of Latent Vectors from Generative Adversarial Networks
• Adversarial Learning for Semi-Supervised Semantic Segmentation
• Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
• Co-Generation with GANs using AIS based HMC
• Controlling generative models with continuous factors of variations

Cited by

• Seg-VAR: Image Segmentation with Visual Autoregressive Modeling
• FRIEREN: Federated Learning with Vision-Language Regularization for Segmentation
• Efficient Generalization via Multimodal Co-Training under Data Scarcity and Distribution Shift
• Dual-supervised Asymmetric Co-training for Semi-supervised Medical Domain Generalization
• IGME: Efficient Chained Method Ensemble for Transferable Semantic Segmentation Attacks
• Multimodal Prototype Alignment for Semi-supervised Pathology Image Segmentation
• UGD-IML: A Unified Generative Diffusion-based Framework for Constrained and Unconstrained Image Manipulation Localization
• Multi-Granularity Feature Calibration via VFM for Domain Generalized Semantic Segmentation
• Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models

Related

• Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network
• Analysing the effectiveness of a generative model for semi-supervised medical image segmentation
• Semantic Segmentation Alternative Technique: Segmentation Domain Generation
• Generative Semantic Segmentation
• Uncertainty-Driven Semantic Segmentation through Human-Machine Collaborative Learning