Multi-Modal Hybrid Architecture for Pedestrian Action Prediction

2020/11/16 by Amir Rasouli, Rasouli, Amir, Tiffany Yau +5 · 8 citations
Computer Science · Engineering · #Architecture #Artificial intelligence #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Control engineering #Engineering #FOS: Computer and information sciences #Feed forward #Geography #Machine learning #Modal #Pedestrian #Pedestrian crossing #Pedestrian detection #Range (aeronautics) #Robotics (cs.RO) #Traffic Prediction and Management Techniques #Transport engineering #Video Surveillance and Tracking Methods #cs.CV #cs.RO

paper · pdf · doi:10.48550/arxiv.2012.00514

published in arXiv (Cornell University) (Cornell University) · 7 pages, 4 Figures, 3 tables, submitted to ICRA 2021

arxiv created 2020/11/16 · openalex publication_date 2020/11/16 · arxiv updated 2020/12/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

Pedestrian behavior prediction is one of the major challenges for intelligent driving systems in urban environments. Pedestrians often exhibit a wide range of behaviors and adequate interpretations of those depend on various sources of information such as pedestrian appearance, states of other road users, the environment layout, etc. To address this problem, we propose a novel multi-modal prediction algorithm that incorporates different sources of information captured from the environment to predict future crossing actions of pedestrians. The proposed model benefits from a hybrid learning architecture consisting of feedforward and recurrent networks for analyzing visual features of the environment and dynamics of the scene. Using the existing 2D pedestrian behavior benchmarks and a newly annotated 3D driving dataset, we show that our proposed model achieves state-of-the-art performance in pedestrian crossing prediction.

Cited by

• ACIT: Attention-Guided Cross-Modal Interaction Transformer for Pedestrian Crossing Intention Prediction
• Multi-Context Fusion Transformer for Pedestrian Crossing Intention Prediction in Urban Environments
• Pedestrian Crossing Intention Prediction Using Multimodal Fusion Network
• Occlusion-Aware Diffusion Model for Pedestrian Intention Prediction
• Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues
• Pedestrian Intention Prediction via Vision-Language Foundation Models
• Pedestrian Intention and Trajectory Prediction in Unstructured Traffic Using IDD-PeD
• Temporal-contextual Event Learning for Pedestrian Crossing Intent Prediction

Related

• PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning
• PePScenes: A Novel Dataset and Baseline for Pedestrian Action Prediction in 3D
• Pedestrian Crossing Intention Prediction Using Multimodal Fusion Network
• Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints
• Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention