View-Invariant Policy Learning via Zero-Shot Novel View Synthesis

2024/09/05 by Stephen Tian, Tian, Stephen, Blake Wulfe +11 · 13 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Internet Traffic Analysis and Secure E-voting #Machine Learning (cs.LG) #Robotics (cs.RO) #Text and Document Classification Technologies

paper · pdf · doi:10.48550/arxiv.2409.03685

openalex publication_date 2024/09/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large-scale visuomotor policy learning is a promising approach toward developing generalizable manipulation systems. Yet, policies that can be deployed on diverse embodiments, environments, and observational modalities remain elusive. In this work, we investigate how knowledge from large-scale visual data of the world may be used to address one axis of variation for generalizable manipulation: observational viewpoint. Specifically, we study single-image novel view synthesis models, which learn 3D-aware scene-level priors by rendering images of the same scene from alternate camera viewpoints given a single input image. For practical application to diverse robotic data, these models must operate zero-shot, performing view synthesis on unseen tasks and environments. We empirically analyze view synthesis models within a simple data-augmentation scheme that we call View Synthesis Augmentation (VISTA) to understand their capabilities for learning viewpoint-invariant policies from single-viewpoint demonstration data. Upon evaluating the robustness of policies trained with our method to out-of-distribution camera viewpoints, we find that they outperform baselines in both simulated and real-world manipulation tasks. Videos and additional visualizations are available at https://s-tian.github.io/projects/vista.

Cited by

• OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning
• See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations
• Invariance Co-training for Robot Visual Generalization
• Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splatting
• Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning
• OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction
• EgoDemoGen: Novel Egocentric Demonstration Generation Enables Viewpoint-Robust Manipulation
• ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation
• Imagination at Inference: Synthesizing In-Hand Views for Robust Visuomotor Policy Inference
• GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Model
• ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representations
• Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)
• Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation

Related

• Multi-View Masked World Models for Visual Robotic Manipulation
• Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning
• DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning
• NVSPolicy: Adaptive Novel-View Synthesis for Generalizable Language-Conditioned Policy Learning
• Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos