2025/09/25 by Junfeng Yan, Yan, Junfeng, Biao Wu +4
Business, Management and Accounting · Computer Science · Engineering · #Autonomous Vehicle Technology and Safety #Business Process Modeling and Analysis #Computation and Language (cs.CL) #F.2.2 #FOS: Computer and information sciences #I.2.7 #Multi-Agent Systems and Negotiation #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2509.21143
openalex publication_date 2025/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Multimodal agents have demonstrated strong performance in general GUI interactions, but their application in automotive systems has been largely unexplored. In-vehicle GUIs present distinct challenges: drivers' limited attention, strict safety requirements, and complex location-based interaction patterns. To address these challenges, we introduce Automotive-ENV, the first high-fidelity benchmark and interaction environment tailored for vehicle GUIs. This platform defines 185 parameterized tasks spanning explicit control, implicit intent understanding, and safety-aware tasks, and provides structured multimodal observations with precise programmatic checks for reproducible evaluation. Building on this benchmark, we propose ASURADA, a geo-aware multimodal agent that integrates GPS-informed context to dynamically adjust actions based on location, environmental conditions, and regional driving norms. Experiments show that geo-aware information significantly improves success on safety-aware tasks, highlighting the importance of location-based context in automotive environments. We will release Automotive-ENV, complete with all tasks and benchmarking tools, to further the development of safe and adaptive in-vehicle agents.