OmniNWM: Omniscient Driving Navigation World Models
2025/10/21 by Bohan Li, Li, Bohan, Zhuang Ma +18 · 3 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2510.18313
openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28
Abstract
Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. However, existing methods are typically restricted to fragmented modality modeling, short-horizon drift, and imprecise action control, while lacking intrinsic mechanisms for policy evaluation. In this paper, we introduce OmniNWM, an Omniscient panoramic Navigation World Model that addresses all three dimensions within a consistent probabilistic framework. For State, OmniNWM generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy, ensuring pixel-level alignment across modalities with joint distribution modeling. To mitigate autoregressive exposure bias, we propose a structured panoramic forcing strategy to stabilize long-horizon generation via stochastic manifold thickening. For Action, we introduce canonical geometric action encoding with normalized panoramic Plücker ray-maps. This representation decouples motion dynamics from sensor intrinsics, enabling precise, zero-shot trajectory control across heterogeneous datasets and camera configurations. For Reward, we derive intrinsic occupancy-grounded dense rewards directly from generated 3D volumes, establishing a reliable closed-loop simulation cycle for evaluating diverse planning agents. Extensive experiments demonstrate that OmniNWM achieves SOTA performance in generation fidelity and control precision, with remarkable zero-shot robustness to novel scenes on NuPlan and in-house datasets with distinct camera rigs. Project page is available at https://arlo0o.github.io/OmniNWM/.
Citations
- Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
- WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World
- Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI
- Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
- Neural Atlas Graphs for Dynamic Scene Decomposition and Editing
- 3D and 4D World Modeling: A Survey
- RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
- Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models
- AD-GS: Object-Aware B-Spline Gaussian Splatting for Self-Supervised Autonomous Driving
- Cameras as Relative Positional Encoding
- World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model
- Epona: Autoregressive Diffusion World Model for Autonomous Driving
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
- ReSim: Reliable World Simulation for Autonomous Driving
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
- Rig3R: Rig-Aware Conditioning for Learned 3D Reconstruction
- Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
- GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control
- GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving
- DiffE2E: Rethinking End-to-End Driving with a Hybrid Action Diffusion and Supervised Policy
- DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving
- FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
- SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning
- PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth
- RayZer: A Self-supervised Large View Synthesis Model
- UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving
- ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representation
- Unraveling the Effects of Synthetic Data on End-to-End Autonomous Driving
- DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation
- UniFuture: A 4D Driving World Model for Future Generation and Perception
- VGGT: Visual Geometry Grounded Transformer
- ReCamMaster: Camera-Controlled Generative Rendering from A Single Video
- CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models
- CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving
- BEVDiffuser: Plug-and-Play Diffusion Model for BEV Denoising with Ground-Truth Guidance
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
- Diffusion-Based Planning for Autonomous Driving with Flexible Guidance
- HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
- DreamDrive: Generative 4D Scene Modeling from Street View Images
- Open-Sora: Democratizing Efficient Video Production for All
- DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
- DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers
- DepthLab: From Partial to Complete
- OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation
- GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
- Doe-1: Closed-Loop Autonomous Driving with Large World Model
- Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Prediction
- UniScene: Unified Occupancy-centric Driving Scene Generation
- UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving
- Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model
- InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models
- Navigation World Models
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving
- Driving View Synthesis on Free-form Trajectories with Generative Prior
- ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration
- MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control
- EMMA: End-to-End Multimodal Model for Autonomous Driving
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
- DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
- Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
- Training-free Camera Control for Video Generation
- CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
- Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control
- M2Depth: Self-supervised Two-Frame Multi-camera Metric Depth Estimation
- OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving
- CameraCtrl: Enabling Camera Control for Text-to-Video Generation
- DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
- Cameras as Rays: Pose Estimation via Ray Diffusion
- DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
- GenAD: Generative End-to-End Autonomous Driving
- MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
- WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
- Panacea: Panoramic and Controllable Video Generation for Autonomous Driving
- OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
- Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans
- Wonder3D: Single Image to 3D using Cross-Domain Diffusion
- MagicDrive: Street View Generation with Diverse 3D Geometry Control
- DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
- Qwen Technical Report
- DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout
- One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception
- OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction
- Sigmoid Loss for Language Image Pre-Training
- Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion
- SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving
- SCPNet: Semantic Scene Completion on Point Cloud
- OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception
- Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction
- Street-View Image Generation from a Bird's-Eye View Layout
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
- Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation
- SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
- MonoScene: Monocular 3D Semantic Scene Completion
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
- DriveGAN: Towards a Controllable High-Quality Neural Simulation
- Learning Transferable Visual Models From Natural Language Supervision
- Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes
- Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion
- Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
- LMSCNet: Lightweight Multiscale 3D Semantic Completion
- TSIT: A Simple and Versatile Framework for Image-to-Image Translation
- Anisotropic Convolutional Networks for 3D Semantic Scene Completion
- PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation
- 3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior
- SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving
- nuScenes: A multimodal dataset for autonomous driving
- Towards Accurate Generative Models of Video: A New Metric & Challenges
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
- Squeeze-and-Excitation Networks
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Cited by
Related