CausNVS: Autoregressive Multi-view Diffusion for Flexible 3D Novel View Synthesis

2025/09/08 by Daniel Watson, Kong, Xin, Yannick Strümpler +6 · 2 citations
Computer Science · Engineering · #Advanced Optical Imaging Technologies #Advanced Vision and Imaging #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2509.06579

openalex publication_date 2025/09/08 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Multi-view diffusion models have shown promise in 3D novel view synthesis, but most existing methods adopt a non-autoregressive formulation. This limits their applicability in world modeling, as they only support a fixed number of views and suffer from slow inference due to denoising all frames simultaneously. To address these limitations, we propose CausNVS, a multi-view diffusion model in an autoregressive setting, which supports arbitrary input-output view configurations and generates views sequentially. We train CausNVS with causal masking and per-frame noise, using pairwise-relative camera pose encodings (CaPE) for precise camera control. At inference time, we combine a spatially-aware sliding-window with key-value caching and noise conditioning augmentation to mitigate drift. Our experiments demonstrate that CausNVS supports a broad range of camera trajectories, enables flexible autoregressive novel view synthesis, and achieves consistently strong visual quality across diverse settings. Project page: https://kxhit.github.io/CausNVS.html.

Citations

• GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
• Stable Virtual Camera: Generative View Synthesis with Diffusion Models
• Continuous 3D Perception Model with Persistent State
• Cosmos World Foundation Model Platform for Physical AI
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
• CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
• DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion
• MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views
• LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
• 3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors
• DepthSplat: Connecting Gaussian Splatting and Depth
• ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
• 3D Reconstruction with Spatial Memory
• Diffusion Models Are Real-Time Game Engines
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
• Controlling Space and Time with Diffusion Models
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
• OpenVLA: An Open-Source Vision-Language-Action Model
• EM Distillation for One-step Diffusion Models
• Diffusion for World Modeling: Visual Details Matter in Atari
• CAT3D: Create Anything in 3D with Multi-View Diffusion Models
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation
• PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis
• EscherNet: A Generative Model for Scalable View Synthesis
• DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision
• DUSt3R: Geometric 3D Vision Made Easy
• MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
• Adversarial Diffusion Distillation
• ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models
• Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
• Generative Novel View Synthesis with 3D-Aware Diffusion Models
• Consistent View Synthesis with Pose-Guided Diffusion Models
• Zero-1-to-3: Zero-shot One Image to 3D Object
• Consistency Models
• DreamFusion: Text-to-3D using 2D Diffusion
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Cascaded Diffusion Models for High Fidelity Image Generation
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
• Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines
• FutureMapping: The Computational Structure of Spatial AI Systems
• Long Short-Term Memory
• A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

Cited by

• WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion
• Positional Encoding Field

Related

• Next-Scale Autoregressive Models are Zero-Shot Single-Image Object View Synthesizers
• ARSS: Taming Decoder-only Autoregressive Visual Generation for View Synthesis From Single View
• Novel View Synthesis with Diffusion Models
• UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis
• Generative Novel View Synthesis with 3D-Aware Diffusion Models