2021/11/30 by Yuval Alaluf, Alaluf, Yuval, Omer Tov +7 · 17 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Encoder #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image (mathematics) #Inference #Inversion (geology) #Machine learning #cs.CV
paper · pdf · doi:10.48550/arxiv.2111.15666
published in arXiv (Cornell University) (Cornell University) · Accepted to CVPR 2022; Project page available at http://yuval-alaluf.github.io/hyperstyle/
openalex publication_date 2021/11/30 · arxiv created 2022/03/29 · arxiv updated 2022/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The inversion of real images into StyleGAN's latent space is a well-studied problem. Nevertheless, applying existing approaches to real-world scenarios remains an open challenge, due to an inherent trade-off between reconstruction and editability: latent space regions which can accurately represent real images typically suffer from degraded semantic control. Recent work proposes to mitigate this trade-off by fine-tuning the generator to add the target image to well-behaved, editable regions of the latent space. While promising, this fine-tuning scheme is impractical for prevalent use as it requires a lengthy training phase for each new image. In this work, we introduce this approach into the realm of encoder-based inversion. We propose HyperStyle, a hypernetwork that learns to modulate StyleGAN's weights to faithfully express a given image in editable regions of the latent space. A naive modulation approach would require training a hypernetwork with over three billion parameters. Through careful network design, we reduce this to be in line with existing encoders. HyperStyle yields reconstructions comparable to those of optimization techniques with the near real-time inference capabilities of encoders. Lastly, we demonstrate HyperStyle's effectiveness on several applications beyond the inversion task, including the editing of out-of-domain images which were never seen during training.