Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

2018/04/09 by Xintao Wang, Wang, Xintao, Ke Yu +5 · 54 citations
Computer Science · #Advanced Image Processing Techniques #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image and Signal Denoising Methods #cs.CV

paper · pdf · doi:10.48550/arxiv.1804.02815

This work is accepted in CVPR 2018. Our project page is http://mmlab.ie.cuhk.edu.hk/projects/SFTGAN/

arxiv created 2018/04/09 · openalex publication_date 2018/04/09 · arxiv updated 2018/04/10 · openalex created_date 2018/04/13 · openalex updated_date 2026/07/28

Abstract

Despite that convolutional neural networks (CNN) have recently demonstrated high-quality reconstruction for single-image super-resolution (SR), recovering natural and realistic texture remains a challenging problem. In this paper, we show that it is possible to recover textures faithful to semantic classes. In particular, we only need to modulate features of a few intermediate layers in a single network conditioned on semantic segmentation probability maps. This is made possible through a novel Spatial Feature Transform (SFT) layer that generates affine transformation parameters for spatial-wise feature modulation. SFT layers can be trained end-to-end together with the SR network using the same loss function. During testing, it accepts an input image of arbitrary size and generates a high-resolution image with just a single forward pass conditioned on the categorical priors. Our final results show that an SR network equipped with SFT can generate more realistic and visually pleasing textures in comparison to state-of-the-art SRGAN and EnhanceNet.

Citations

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Modulating early visual processing by language
• MemNet: A Persistent Memory Network for Image Restoration
• Fully Convolutional Networks for Semantic Segmentation

Cited by

• An Integrated Deep Learning and Statistical Framework for Whole-Network Gene--Environment Association with Leaf Vascular Architecture
• Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networks
• FMA-Net++: Motion- and Exposure-Aware Real-World Joint Video Super-Resolution and Deblurring
• MRI Super-Resolution with Deep Learning: A Comprehensive Survey
• Segmentation-Aware Latent Diffusion for Satellite Image Super-Resolution: Enabling Smallholder Farm Boundary Delineation
• BasicAVSR: Arbitrary-Scale Video Super-Resolution via Image Priors and Enhanced Motion Compensation
• Semantic-Guided Cross-Sensor Super Resolution of Remote Sensing Images: A Gated Dual Conditioning Flow Matching Model
• SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning
• BrainPuzzle: Hybrid Physics and Data-Driven Reconstruction for Transcranial Ultrasound Tomography
• Efficient Perceptual Image Super Resolution: AIM 2025 Study and Benchmark
• AnyUp: Universal Feature Upsampling
• Interactive Atmospheric Composition Emulation for Next-Generation Earth System Models
• Enabling High-Quality In-the-Wild Imaging from Severely Aberrated Metalens Bursts
• HeadsUp! High-Fidelity Portrait Image Super-Resolution
• SDAKD: Student Discriminator Assisted Knowledge Distillation for Super-Resolution Generative Adversarial Networks
• Unleashing the Potential of the Semantic Latent Space in Diffusion Models for Image Dehazing
• Deep Learning Empowered Super-Resolution: A Comprehensive Survey and Future Prospects
• Dataset Distillation for Super-Resolution without Class Labels and Pre-trained Models
• EatGAN: An Edge-Attention Guided Generative Adversarial Network for Single Image Super-Resolution
• RAM++: Robust Representation Learning via Adaptive Mask for All-in-One Image Restoration
• BIR-Adapter: A Low-Complexity Diffusion Model Adapter for Blind Image Restoration
• HumanGenesis: Agent-Based Geometric and Generative Modeling for Synthetic Human Dynamics
• Enhanced Generative Structure Prior for Chinese Text Image Super-resolution
• Unsupervised Real-World Super-Resolution via Rectified Flow Degradation Modelling
• Exploring Probabilistic Modeling Beyond Domain Generalization for Semantic Segmentation
• Efficient Burst Super-Resolution with One-step Diffusion
• Diffusion-Guided Knowledge Distillation for Weakly-Supervised Low-Light Semantic Segmentation
• Customizable ROI-Based Deep Image Compression
• Metadata, Wavelet, and Time Aware Diffusion Models for Satellite Image Super Resolution
• Progressive Alignment Degradation Learning for Pansharpening
• RealSR-R1: Reinforcement Learning for Real-World Image Super-Resolution with Vision-Language Chain-of-Thought
• Self-Cascaded Diffusion Models for Arbitrary-Scale Image Super-Resolution
• SuperMeshing: A New Deep Learning Architecture for Increasing the Mesh Density of Metal Forming Stress Field with Attention Mechanism and Perceptual Features
• Reconstructing High-resolution Turbulent Flows Using Physics-Guided Neural Networks
• Controlled Data Rebalancing in Multi-Task Learning for Real-World Image Super-Resolution
• Multi-scale Image Super Resolution with a Single Auto-Regressive Model
• Unsupervised Degradation Learning for Single Image Super-Resolution
• Fixed-Length Dense Fingerprint Representation with Alignment and Robust Enhancement
• Optimization of Module Transferability in Single Image Super-Resolution: Universality Assessment and Cycle Residual Blocks
• Channel-wise and Spatial Feature Modulation Network for Single Image Super-Resolution
• Instance Data Condensation for Image Super-Resolution
• Deep Network Interpolation for Continuous Imagery Effect Transition
• Detail Continuation over a Trustworthy Coarse Scale for Autoregressive Super-Resolution
• Efficient Degradation-agnostic Image Restoration via Channel-Wise Functional Decomposition and Manifold Regularization
• SuperPure: Efficient Purification of Localized and Distributed Adversarial Patches via Super-Resolution GAN Models
• Towards a Universal Image Degradation Model via Content-Degradation Disentanglement
• Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
• EAM: Enhancing Anything with Diffusion Transformers for Blind Super-Resolution
• Quaternion Wavelet-Conditioned Diffusion Models for Image Super-Resolution
• Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform
• NTIRE 2025 Challenge on Short-form UGC Video Quality Assessment and Enhancement: KwaiSR Dataset and Study
• Any Image Restoration via Efficient Spatial-Frequency Degradation Adaptation
• High-Fidelity Image Inpainting with Multimodal Guided GAN Inversion
• NTIRE 2025 Challenge on Short-form UGC Video Quality Assessment and Enhancement: Methods and Results

Related

• EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
• Learning Texture Transformer Network for Image Super-Resolution
• The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution
• Natural and Realistic Single Image Super-Resolution with Explicit Natural Manifold Discrimination
• Image Super-Resolution Using a Wavelet-based Generative Adversarial Network