Semantic Image Synthesis with Spatially-Adaptive Normalization

2019/03/31 by Taesung Park, Ming-Yu Liu, Ting-Chun Wang +1 · 4 citations
Computer Science · #cs.CV #cs.AI #cs.GR #cs.LG

paper · pdf

published as CVPR 2019 · Accepted as a CVPR 2019 oral paper

arxiv created 2019/11/05 · arxiv updated 2019/11/06

Abstract

We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to ``wash away'' semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style. Code is available at https://github.com/NVlabs/SPADE .

Cited by

• Manifold Induced Biases for Zero-shot and Few-shot Detection of Generated Images
• MLEP: Multi-granularity Local Entropy Patterns for Universal AI-generated Image Detection
• Hadamard product in deep learning: Introduction, Advances and Challenges
• High-Fidelity Image Inpainting with Multimodal Guided GAN Inversion