RUST: Latent Neural Scene Representations from Unposed Imagery

2022/11/25 by Mehdi S. M. Sajjadi, Sajjadi, Mehdi S. M., Aravindh Mahendran +11 · 6 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Graphics (cs.GR) #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Robotics and Sensor-Based Localization #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2211.14306

openalex publication_date 2022/11/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Inferring the structure of 3D scenes from 2D observations is a fundamental challenge in computer vision. Recently popularized approaches based on neural scene representations have achieved tremendous impact and have been applied across a variety of applications. One of the major remaining challenges in this space is training a single model which can provide latent representations which effectively generalize beyond a single scene. Scene Representation Transformer (SRT) has shown promise in this direction, but scaling it to a larger set of diverse scenes is challenging and necessitates accurately posed ground truth data. To address this problem, we propose RUST (Really Unposed Scene representation Transformer), a pose-free approach to novel view synthesis trained on RGB images alone. Our main insight is that one can train a Pose Encoder that peeks at the target image and learns a latent pose embedding which is used by the decoder for view synthesis. We perform an empirical investigation into the learned latent pose structure and show that it allows meaningful test-time camera transformations and accurate explicit pose readouts. Perhaps surprisingly, RUST achieves similar quality as methods which have access to perfect camera pose, thereby unlocking the potential for large-scale training of amortized neural scene representations.

Cited by

• ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training
• E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training
• Efficiently Reconstructing Dynamic Scenes One D4RT at a Time
• True Self-Supervised Novel View Synthesis is Transferable
• Attention over Scene Graphs: Indoor Scene Representations Toward CSAI Classification
• RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video

Related

• Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformers
• Neural Scene Graphs for Dynamic Scenes
• Inferring 3D Object Pose in RGB-D Images
• NViST: In the Wild New View Synthesis from a Single Image with Transformers
• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations