Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation

2021/08/05 by Duo Peng, Peng, Duo, Yinjie Lei +7 · 16 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #cs.CV

paper · pdf · doi:10.48550/arxiv.2108.02376

15 pages, 14 figures, accepted by IEEE Transactions on Image Processing (TIP 2021)

openalex publication_date 2021/08/05 · arxiv created 2021/08/06 · arxiv updated 2021/08/09 · openalex created_date 2021/08/16 · openalex updated_date 2026/07/28

Abstract

Semantic segmentation is a crucial image understanding task, where each pixel of image is categorized into a corresponding label. Since the pixel-wise labeling for ground-truth is tedious and labor intensive, in practical applications, many works exploit the synthetic images to train the model for real-word image semantic segmentation, i.e., Synthetic-to-Real Semantic Segmentation (SRSS). However, Deep Convolutional Neural Networks (CNNs) trained on the source synthetic data may not generalize well to the target real-world data. In this work, we propose two simple yet effective texture randomization mechanisms, Global Texture Randomization (GTR) and Local Texture Randomization (LTR), for Domain Generalization based SRSS. GTR is proposed to randomize the texture of source images into diverse unreal texture styles. It aims to alleviate the reliance of the network on texture while promoting the learning of the domain-invariant cues. In addition, we find the texture difference is not always occurred in entire image and may only appear in some local areas. Therefore, we further propose a LTR mechanism to generate diverse local regions for partially stylizing the source images. Finally, we implement a regularization of Consistency between GTR and LTR (CGL) aiming to harmonize the two proposed mechanisms during training. Extensive experiments on five publicly available datasets (i.e., GTA5, SYNTHIA, Cityscapes, BDDS and Mapillary) with various SRSS settings (i.e., GTA5/SYNTHIA to Cityscapes/BDDS/Mapillary) demonstrate that the proposed method is superior to the state-of-the-art methods for domain generalization based SRSS.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Domain Generalization via Invariant Feature Representation
• ImageNet classification with deep convolutional neural networks
• Unsupervised Domain Adaptation with Residual Transfer Networks
• SceneNet: Understanding Real World Indoor Scenes With Synthetic Data
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Instance Normalization: The Missing Ingredient for Fast Stylization
• FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Improved Regularization of Convolutional Neural Networks with Cutout
• mixup: Beyond Empirical Risk Minimization
• Decorrelated Batch Normalization
• Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
• ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
• See More Than Once -- Kernel-Sharing Atrous Convolution for Semantic Segmentation
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• Test-Time Modification: Inverse Domain Transformation for Robust Perception
• DistillFSS: Synthesizing Few-Shot Knowledge into a Lightweight Segmentation Model
• SAGE: Style-Adaptive Generalization for Privacy-Constrained Semantic Segmentation Across Domains
• Do We Need Perfect Data? Leveraging Noise for Domain Generalized Segmentation
• Domain Generalization for Semantic Segmentation: A Survey
• FRIEREN: Federated Learning with Vision-Language Regularization for Segmentation
• SynthGenNet: a self-supervised approach for test-time generalization using synthetic multi-source domain mixing of street view images
• IELDG: Suppressing Domain-Specific Noise with Inverse Evolution Layers for Domain Generalized Semantic Segmentation
• Multi-Granularity Feature Calibration via VFM for Domain Generalized Semantic Segmentation
• Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models
• Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models
• Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentation
• Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation
• LangDAug: Langevin Data Augmentation for Multi-Source Domain Generalization in Medical Image Segmentation
• Depth-Sensitive Soft Suppression with RGB-D Inter-Modal Stylization Flow for Domain Generalization Semantic Segmentation
• What is the Added Value of UDA in the VFM Era?
• Stronger, Steadier & Superior: Geometric Consistency in Depth VFM Forges Domain Generalized Semantic Segmentation

Related

• Texture Learning Domain Randomization for Domain Generalized Segmentation
• Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation
• Learning Statistical Texture for Semantic Segmentation
• Texture Underfitting for Domain Adaptation
• Adaptive Texture Filtering for Single-Domain Generalized Segmentation