2024/09/30 by Bahri Batuhan Bilecen, Bilecen, Bahri Batuhan, Ahmet Berke Gokmen +3 · 2 citations
Earth and Planetary Sciences · Medicine · #Advanced MRI Techniques and Applications #Computational Geometry (cs.CG) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Graphics (cs.GR) #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Medical Imaging Techniques and Applications #Seismic Imaging and Inversion Techniques #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2409.20530
openalex publication_date 2024/09/30 · openalex created_date 2024/10/28 · openalex updated_date 2026/07/28
3D GAN inversion aims to project a single image into the latent space of a 3D Generative Adversarial Network (GAN), thereby achieving 3D geometry reconstruction. While there exist encoders that achieve good results in 3D GAN inversion, they are predominantly built on EG3D, which specializes in synthesizing near-frontal views and is limiting in synthesizing comprehensive 3D scenes from diverse viewpoints. In contrast to existing approaches, we propose a novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspective. To achieve realistic 3D modeling of the input image, we introduce a dual encoder system tailored for high-fidelity reconstruction and realistic generation from different viewpoints. Accompanying this, we propose a stitching framework on the triplane domain to get the best predictions from both. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-aware triplane discriminator. Experiments reveal that our approach surpasses the existing encoder training methods qualitatively and quantitatively. Please visit the project page: https://berkegokmen1.github.io/dual-enc-3d-gan-inv.