Multi-style Generative Network for Real-time Transfer

2017/03/20 by Hang Zhang, Zhang, Hang, Kristin Dana +1 · 113 citations
Computer Science · Mathematics · #Advanced Image Processing Techniques #Advanced Vision and Imaging #Artificial intelligence #Artificial neural network #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Convolution (computer science) #FOS: Computer and information sciences #Flexibility (engineering) #Generative Adversarial Networks and Image Synthesis #Image (mathematics) #Interpolation (computer graphics) #Mathematics #Quality (philosophy) #Style (visual arts) #Upsampling #cs.CV

paper · pdf · doi:10.48550/arxiv.1703.06953

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2017/03/20 · arxiv created 2017/11/16 · arxiv updated 2017/11/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the rapid progress in style transfer, existing approaches using feed-forward generative network for multi-style or arbitrary-style transfer are usually compromised of image quality and model flexibility. We find it is fundamentally difficult to achieve comprehensive style modeling using 1-dimensional style embedding. Motivated by this, we introduce CoMatch Layer that learns to match the second order feature statistics with the target styles. With the CoMatch Layer, we build a Multi-style Generative Network (MSG-Net), which achieves real-time performance. We also employ an specific strategy of upsampled convolution which avoids checkerboard artifacts caused by fractionally-strided convolution. Our method has achieved superior image quality comparing to state-of-the-art approaches. The proposed MSG-Net as a general approach for real-time style transfer is compatible with most existing techniques including content-style interpolation, color-preserving, spatial control and brush stroke size control. MSG-Net is the first to achieve real-time brush-size control in a purely feed-forward manner for style transfer. Our implementations and pre-trained models for Torch, PyTorch and MXNet frameworks will be publicly available.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ImageNet Large Scale Visual Recognition Challenge
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Mode Regularized Generative Adversarial Networks
• Deep Residual Learning for Image Recognition
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Fully Convolutional Networks for Semantic Segmentation

Cited by

• Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulation
• MetaStyle: Three-Way Trade-Off Among Speed, Flexibility, and Quality in Neural Style Transfer
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• Neural Style Transfer: A Review
• Autonomy, Authenticity, Authorship and Intention in computer generated art
• Context Encoding for Semantic Segmentation
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• Learning to Transfer Visual Effects from Videos to Images
• Spatially Controllable Image Synthesis with Internal Representation Collaging
• Dynamic Instance Normalization for Arbitrary Style Transfer
• Meta Networks for Neural Style Transfer
• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
• Language-Driven Image Style Transfer
• DRB-GAN: A Dynamic ResBlock Generative Adversarial Network for Artistic Style Transfer
• CoCoPIE: Making Mobile AI Sweet As PIE --Compression-Compilation Co-Design Goes a Long Way
• Generative Adversarial Network-based Synthesis of Visible Faces from Polarimetric Thermal Faces
• Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network
• Density-aware Single Image De-raining using a Multi-stream Dense Network
• Deblurring Face Images Using Uncertainty Guided Multi-Stream Semantic Networks

Related

• Multiple Style-Transfer in Real-Time
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
• Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
• Multimodal Style Transfer via Graph Cuts
• MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference