Explainable Action Prediction through Self-Supervision on Scene Graphs

2023/02/07 by Pawit Kochakarn, Daniele De Martini, Kochakarn, Pawit +5 · 2 citations
Computer Science · Engineering · #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning in Healthcare #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2302.03477

openalex publication_date 2023/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work explores scene graphs as a distilled representation of high-level information for autonomous driving, applied to future driver-action prediction. Given the scarcity and strong imbalance of data samples, we propose a self-supervision pipeline to infer representative and well-separated embeddings. Key aspects are interpretability and explainability; as such, we embed in our architecture attention mechanisms that can create spatial and temporal heatmaps on the scene graphs. We evaluate our system on the ROAD dataset against a fully-supervised approach, showing the superiority of our training regime.

Cited by

• Hierarchical Feature-level Reverse Propagation for Post-Training Neural Networks
• Leveraging Foundation Models for Multimodal Graph-Based Action Recognition

Related

• Towards Consistent and Explainable Motion Prediction using Heterogeneous Graph Attention
• Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Driving
• GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving
• Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving
• Just Label What You Need: Fine-Grained Active Selection for Perception and Prediction through Partially Labeled Scenes