DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction

2025/06/28 by Zhiyi Hou, Enhui Ma, Hou, Zhiyi +26 · 1 voice · 2 citations
Computer Science · Engineering · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #I.2.10 #I.2.7 #I.4.8 #Multimodal Machine Learning Applications #Robotics (cs.RO) #cs.AI #cs.CV #cs.RO

paper · pdf · doi:10.48550/arxiv.2507.02948

openalex publication_date 2025/06/28 · arxiv published 2025/06/28 · arxiv updated 2025/07/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Autonomous driving has seen significant progress, driven by extensive real-world data. However, in long-tail scenarios, accurately predicting the safety of the ego vehicle's future motion remains a major challenge due to uncertainties in dynamic environments and limitations in data coverage. In this work, we aim to explore whether it is possible to enhance the motion risk prediction capabilities of Vision-Language Models (VLM) by synthesizing high-risk motion data. Specifically, we introduce a Bird's-Eye View (BEV) based motion simulation method to model risks from three aspects: the ego-vehicle, other vehicles, and the environment. This allows us to synthesize plug-and-play, high-risk motion data suitable for VLM training, which we call DriveMRP-10K. Furthermore, we design a VLM-agnostic motion risk estimation framework, named DriveMRP-Agent. This framework incorporates a novel information injection strategy for global context, ego-vehicle perspective, and trajectory projection, enabling VLMs to effectively reason about the spatial relationships between motion waypoints and the environment. Extensive experiments demonstrate that by fine-tuning with DriveMRP-10K, our DriveMRP-Agent framework can significantly improve the motion risk prediction performance of multiple VLM baselines, with the accident recognition accuracy soaring from 27.13% to 88.03%. Moreover, when tested via zero-shot evaluation on an in-house real-world high-risk motion dataset, DriveMRP-Agent achieves a significant performance leap, boosting the accuracy from base_model's 29.42% to 68.50%, which showcases the strong generalization capabilities of our method in real-world scenarios.

Citations

• Comparison of Waymo Rider-Only Crash Rates by Crash Type to Human Benchmarks at 56.7 Million Miles
• Qwen2.5-VL Technical Report
• Autonomous Driving in Unstructured Environments: How Far Have We Come?
• Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning
• ComDrive: Comfort-Oriented End-to-End Autonomous Driving
• The Llama 3 Herd of Models
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
• Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
• DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• DriveLM: Driving with Graph Visual Question Answering
• Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
• Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
• Improved Baselines with Visual Instruction Tuning
• GPT-Driver: Learning to Drive with GPT
• Safe Navigation in Unstructured Environments by Minimizing Uncertainty in Control and Perception
• VAD: Vectorized Scene Representation for Efficient Autonomous Driving
• GPT-4 Technical Report
• Planning-oriented Autonomous Driving
• MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
• NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
• BERTScore: Evaluating Text Generation with BERT
• nuScenes: A multimodal dataset for autonomous driving
• CARLA: An Open Urban Driving Simulator
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
• S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking

Cited by

• Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
• NSF-HRPT: Neural Semantic Field meets Hierarchical Risk Perception Tree for Safety-Critical Scenario Assessment

Discussions

• DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction arxiv: arxiv.org/abs/2507.02948 code: github.com/hzy138/Drive... [bsky, 0 points, 0 comments]

Related

• Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM
• Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models
• INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluation
• EDRF: Enhanced Driving Risk Field Based on Multimodal Trajectory Prediction and Its Applications
• Addressing Corner Cases in Autonomous Driving: A World Model-based Approach with Mixture of Experts and LLMs