Consistent View Synthesis with Pose-Guided Diffusion Models

2023/03/30 by Tseng, Hung-Yu, Li, Qinbo, Kim, Changil +3 · 14 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2303.17598

Abstract

Novel view synthesis from a single image has been a cornerstone problem for many Virtual Reality applications that provide immersive experiences. However, most existing techniques can only synthesize novel views within a limited range of camera motion or fail to generate consistent and high-quality novel views under significant camera movement. In this work, we propose a pose-guided diffusion model to generate a consistent long-term video of novel views from a single image. We design an attention layer that uses epipolar lines as constraints to facilitate the association between different viewpoints. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed diffusion model against state-of-the-art transformer-based and GAN-based approaches.

Cited by

• ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
• SUCCESS-GS: Survey of Compactness and Compression for Efficient Static and Dynamic Gaussian Splatting
• MVRoom: Controllable 3D Indoor Scene Generation with Multi-View Diffusion Models
• CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Models
• Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis
• Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Image
• MagicView: Multi-View Consistent Identity Customization via Priors-Guided In-Context Learning
• CausNVS: Autoregressive Multi-view Diffusion for Flexible 3D Novel View Synthesis
• Look Beyond: Two-Stage Scene View Generation via Panorama and Video Diffusion
• DMS:Diffusion-Based Multi-Baseline Stereo Generation for Improving Self-Supervised Depth Estimation
• Novel View Synthesis using DDIM Inversion
• StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
• DOMR: Establishing Cross-View Segmentation via Dense Object Matching
• Veila: Panoramic LiDAR Generation from a Monocular RGB Image

Related

• ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models
• Look Beyond: Two-Stage Scene View Generation via Panorama and Video Diffusion
• Stable Virtual Camera: Generative View Synthesis with Diffusion Models
• ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
• Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models