Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments

2022/04/20 by Jacob Krantz, Krantz, Jacob, Stefan Lee +1 · 28 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Robotics (cs.RO) #cs.CL #cs.CV #cs.RO

paper · pdf · doi:10.48550/arxiv.2204.09667

Changes: figure compression for accessibility

openalex publication_date 2022/04/20 · arxiv created 2022/04/24 · arxiv updated 2022/04/26 · openalex created_date 2022/04/26 · openalex updated_date 2026/07/28

Abstract

Recent work in Vision-and-Language Navigation (VLN) has presented two environmental paradigms with differing realism -- the standard VLN setting built on topological environments where navigation is abstracted away, and the VLN-CE setting where agents must navigate continuous 3D environments using low-level actions. Despite sharing the high-level task and even the underlying instruction-path data, performance on VLN-CE lags behind VLN significantly. In this work, we explore this gap by transferring an agent from the abstract environment of VLN to the continuous environment of VLN-CE. We find that this sim-2-sim transfer is highly effective, improving over the prior state of the art in VLN-CE by +12% success rate. While this demonstrates the potential for this direction, the transfer does not fully retain the original performance of the agent in the abstract setting. We present a sequence of experiments to identify what differences result in performance degradation, providing clear directions for further improvement.

Cited by

• MemVLN: Episodic and Procedural Memory for Vision-and-Language Navigation
• Towards Dual-Brain Minimal Sufficient Representation for Vision-Language Navigation
• Structured Observation Language for Efficient and Generalizable Vision-Language Navigation
• AstraNav-World: World Model for Foresight Control and Consistency
• Efficient-VLN: A Training-Efficient Vision-Language Navigation Model
• NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction
• Language-conditioned world model improves policy generalization by reading environmental descriptions
• MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robots
• OpenVLN: Open-world Aerial Vision-Language Navigation
• Fast-SmartWay: Panoramic-Free End-to-End Zero-Shot Vision-and-Language Navigation
• NavSpace: How Navigation Agents Follow Spatial Intelligence Instructions
• OmniNav: A Unified Framework for Prospective Exploration and Visual-Language Navigation
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation
• Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Prompting
• VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigation
• Embodied Navigation Foundation Model
• GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigation
• CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Model
• DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigation
• Harnessing Input-Adaptive Inference for Efficient VLN
• Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparities
• VPN: Visual Prompt Navigation
• StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
• View Invariant Learning for Vision-Language Navigation in Continuous Environments
• NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
• Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation
• CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memory
• ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision-and-Language Navigation in Continuous Environments

Related

• Sim-to-Real Transfer for Vision-and-Language Navigation
• Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation
• Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions
• Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
• Scaling Data Generation in Vision-and-Language Navigation