2020/11/24 by Michael Niemeyer, Andreas Geiger, Niemeyer, Michael +1 · 27 citations
Computer Science · #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Processing and 3D Reconstruction #Machine Learning (cs.LG) #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.2011.12100
Accepted to CVPR 2021 (oral). Project page: http://bit.ly/giraffe-project
openalex publication_date 2020/11/24 · arxiv created 2021/04/29 · arxiv updated 2021/04/30 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Deep generative models allow for photorealistic image synthesis at high resolutions. But for many applications, this is not enough: content creation also needs to be controllable. While several recent works investigate how to disentangle underlying factors of variation in the data, most of them operate in 2D and hence ignore that our world is three-dimensional. Further, only few works consider the compositional nature of scenes. Our key hypothesis is that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis. Representing scenes as compositional generative neural feature fields allows us to disentangle one or multiple objects from the background as well as individual objects' shapes and appearances while learning from unstructured and unposed image collections without any additional supervision. Combining this scene representation with a neural rendering pipeline yields a fast and realistic image synthesis model. As evidenced by our experiments, our model is able to disentangle individual objects and allows for translating and rotating them in the scene as well as changing the camera pose.