Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving

2024/03/17 by Xiaoji Zheng, Lixiu Wu, Zheng, Xiaoji +13 · 5 citations
Computer Science · #68T45 #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2403.11057

openalex publication_date 2024/03/17 · openalex created_date 2024/03/21 · openalex updated_date 2026/07/28

Abstract

Motion prediction is among the most fundamental tasks in autonomous driving. Traditional methods of motion forecasting primarily encode vector information of maps and historical trajectory data of traffic participants, lacking a comprehensive understanding of overall traffic semantics, which in turn affects the performance of prediction tasks. In this paper, we utilized Large Language Models (LLMs) to enhance the global traffic context understanding for motion prediction tasks. We first conducted systematic prompt engineering, visualizing complex traffic environments and historical trajectory information of traffic participants into image prompts -- Transportation Context Map (TC-Map), accompanied by corresponding text prompts. Through this approach, we obtained rich traffic context information from the LLM. By integrating this information into the motion prediction model, we demonstrate that such context can enhance the accuracy of motion predictions. Furthermore, considering the cost associated with LLMs, we propose a cost-effective deployment strategy: enhancing the accuracy of motion prediction tasks at scale with 0.7% LLM-augmented datasets. Our research offers valuable insights into enhancing the understanding of traffic scenes of LLMs and the motion prediction performance of autonomous driving. The source code is available at \urlhttps://github.com/AIR-DISCOVER/LLM-Augmented-MTR and \urlhttps://aistudio.baidu.com/projectdetail/7809548.

Cited by

• SVBRD-LLM: Self-Verifying Behavioral Rule Discovery for Autonomous Vehicle Identification
• LongComp: Long-Tail Compositional Zero-Shot Generalization for Robust Trajectory Prediction
• SafeMT: Multi-turn Safety for Multimodal Language Models
• Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
• Delving into Mapping Uncertainty for Mapless Trajectory Prediction

Related

• CoT-Drive: Efficient Motion Forecasting for Autonomous Driving with LLMs and Chain-of-Thought Prompting
• Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
• Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models
• Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models
• Large Trajectory Models are Scalable Motion Predictors and Planners