Globally and locally consistent image completion

2017/07/20 by Satoshi Iizuka, Edgar Simo-Serra, Edgar Simo‐Serra +1 · 49 citations
Computer Science · Engineering · #Advanced Image Processing Techniques #Generative Adversarial Networks and Image Synthesis #Image Processing Techniques and Applications

paper · doi:10.1145/3072959.3073659

openalex publication_date 2017/07/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

We present a novel approach for image completion that results in images that are both locally and globally consistent. With a fully-convolutional neural network, we can complete images of arbitrary resolutions by filling-in missing regions of any shape. To train this image completion network to be consistent, we use global and local context discriminators that are trained to distinguish real images from completed ones. The global discriminator looks at the entire image to assess if it is coherent as a whole, while the local discriminator looks only at a small area centered at the completed region to ensure the local consistency of the generated patches. The image completion network is then trained to fool the both context discriminator networks, which requires it to generate images that are indistinguishable from real ones with regard to overall consistency as well as in details. We show that our approach can be used to complete a wide variety of scenes. Furthermore, in contrast with the patch-based approaches such as PatchMatch, our approach can generate fragments that do not appear elsewhere in the image, which allows us to naturally complete the images of objects with familiar and highly specific structures, such as faces.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Backpropagation Applied to Handwritten Zip Code Recognition
• Improved Techniques for Training GANs
• Places: An Image Database for Deep Scene Understanding
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Cited by

• Towards High-Level Semantic Intelligence
• Disturbance-Free Surgical Video Generation from Multi-Camera Shadowless Lamps for Open Surgery
• Semantic-Guided Two-Stage GAN for Face Inpainting with Hybrid Perceptual Encoding
• Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detection
• Generative deep learning for foundational video translation in ultrasound
• GeoComplete: Geometry-Aware Diffusion for Reference-Driven Image Completion
• LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal
• Everybody's Talkin': Let Me Talk as You Want
• Painting Outside the Box: Image Outpainting with GANs
• Chest X-ray Inpainting with Deep Generative Models
• DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality
• Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image
• Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE
• Unpaired Image-to-Image Translation using Adversarial Consistency Loss
• Learning to Incorporate Structure Knowledge for Image Inpainting
• SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches
• Hierarchical Regression Network for Spectral Reconstruction from RGB Images
• Foreground-aware Image Inpainting
• RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal
• The Impact of Hole Geometry on Relative Robustness of In-Painting Networks: An Empirical Study
• Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation
• Semi-parametric Image Synthesis
• Pixel-wise Dense Detector for Image Inpainting
• Label-Noise Robust Generative Adversarial Networks
• IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed
• Very Long Natural Scenery Image Prediction by Outpainting
• An Internal Learning Approach to Video Inpainting
• The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks
• Diverse Image Inpainting with Bidirectional and Autoregressive Transformers
• SplatFill: 3D Scene Inpainting via Depth-Guided Gaussian Splatting
• Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology
• Improving Consistency and Correctness of Sequence Inpainting using\n Semantically Guided Generative Adversarial Network
• Texture and geometric feature-fusion-based network for Dunhuang mural inpainting
• Fast Generation of High Fidelity RGB-D Images by Deep-Learning with Adaptive Convolution
• Optimizing Paths for Adaptive Fly-Scan Microscopy: An Extended Version
• Blur-Attention: A boosting mechanism for non-uniform blurred image restoration
• WaveFill: A Wavelet-based Generation Network for Image Inpainting
• Towards Real-World Blind Face Restoration with Generative Facial Prior
• Coherent Semantic Attention for Image Inpainting
• ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
• Visual-Neural-Inspired Image Inpainting for Specific Objects-of-Interest Imaging
• Progressive Temporal Feature Alignment Network for Video Inpainting
• Image Inpainting via Generative Multi-column Convolutional Neural Networks
• DVI: Depth Guided Video Inpainting for Autonomous Driving
• MILD: Multi-Layer Diffusion Strategy for Complex and Precise Multi-IP Aware Human Erasing
• IN2OUT: Fine-Tuning Video Inpainting Model for Video Outpainting Using Hierarchical Discriminator
• Environment reconstruction on depth images using Generative Adversarial\n Networks
• Deep Likelihood Network for Image Restoration with Multiple Degradation Levels

Related

• Image Completion and Extrapolation with Contextual Cycle Consistency
• Distance Weighted Trans Network for Image Completion
• Pluralistic Image Completion
• Generative Face Completion
• Contour-guided Image Completion with Perceptual Grouping