Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network

2018/02/26 by Zizhao Zhang, Zhang, Zizhao, Yuanpu Xie +3 · 2 citations
Computer Science · #Advanced Image Processing Techniques #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #cs.CV

paper · pdf · doi:10.48550/arxiv.1802.09178

CVPR2018 Spotlight

openalex publication_date 2018/02/26 · openalex created_date 2018/03/06 · arxiv created 2018/04/06 · arxiv updated 2018/04/10 · openalex updated_date 2026/07/28

Abstract

This paper presents a novel method to deal with the challenging task of generating photographic images conditioned on semantic image descriptions. Our method introduces accompanying hierarchical-nested adversarial objectives inside the network hierarchies, which regularize mid-level representations and assist generator training to capture the complex image statistics. We present an extensile single-stream generator architecture to better adapt the jointed discriminators and push generated images up to high resolutions. We adopt a multi-purpose adversarial loss to encourage more effective image and text information usage in order to improve the semantic consistency and image fidelity simultaneously. Furthermore, we introduce a new visual-semantic similarity measure to evaluate the semantic consistency of generated images. With extensive experimental validation on three public datasets, our method significantly improves previous state of the arts on all datasets over different evaluation metrics.

Citations

• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Adam: A Method for Stochastic Optimization
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Improved Techniques for Training GANs
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Conditional Image Synthesis With Auxiliary Classifier GANs
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
• Semantic Image Synthesis via Adversarial Learning
• Dual Discriminator Generative Adversarial Nets
• Adversarial nets with perceptual losses for text-to-image synthesis
• Least Squares Generative Adversarial Networks
• Multi-style Generative Network for Real-time Transfer
• Going Deeper with Convolutions
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Pyramid Scene Parsing Network
• Fully Convolutional Networks for Semantic Segmentation

Cited by

• Densely Connected Pyramid Dehazing Network
• MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination

Related

• StackGAN: Text to Photo-realistic Image Synthesis with Stacked\n Generative Adversarial Networks
• Photographic Image Synthesis with Cascaded Refinement Networks
• Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks
• Image Synthesis with Adversarial Networks: a Comprehensive Survey and Case Studies
• Dual Pyramid Generative Adversarial Networks for Semantic Image Synthesis