Latent Uncertainty Representations for Video-based Driver Action and Intention Recognition

2025/10/06 by Koen Vellenga, Vellenga, Koen, H. Joe Steinhauer +5
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2510.05006

openalex publication_date 2025/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep neural networks (DNNs) are increasingly applied to safety-critical tasks in resource-constrained environments, such as video-based driver action and intention recognition. While last layer probabilistic deep learning (LL-PDL) methods can detect out-of-distribution (OOD) instances, their performance varies. As an alternative to last layer approaches, we propose extending pre-trained DNNs with transformation layers to produce multiple latent representations to estimate the uncertainty. We evaluate our latent uncertainty representation (LUR) and repulsively trained LUR (RLUR) approaches against eight PDL methods across four video-based driver action and intention recognition datasets, comparing classification performance, calibration, and uncertainty-based OOD detection. We also contribute 28,000 frame-level action labels and 1,194 video-level intention labels for the NuScenes dataset. Our results show that LUR and RLUR achieve comparable in-distribution classification performance to other LL-PDL approaches. For uncertainty-based OOD detection, LUR matches top-performing PDL methods while being more efficient to train and easier to tune than approaches that require Markov-Chain Monte Carlo sampling or repulsive training procedures.

Citations

• Last Layer Hamiltonian Monte Carlo
• Aleatoric and Epistemic Uncertainty Measures for Ordinal Classification through Binary Reduction
• Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles
• Variational Bayesian Last Layers
• Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks
• Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI
• Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models
• Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes
• NeuRAD: Neural Rendering for Autonomous Driving
• A Primer on Bayesian Neural Networks: Review and Debates
• Out-of-Distribution Detection for Monocular Depth Estimation
• AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception
• E2E-LOAD: End-to-End Long-form Online Action Detection
• CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers
• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference
• Packed-Ensembles for Efficient Uncertainty Estimation
• GateHUB: Gated History Unit with Background Suppression for Online Action Detection
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
• Three things everyone should know about Vision Transformers
• A Systematic Review of Robustness in Deep Learning for Computer Vision: Mind the gap?
• A Survey of Uncertainty in Deep Neural Networks
• Long Short-Term Transformer for Online Action Detection
• On the Practicality of Deterministic Epistemic Uncertainty
• Repulsive Deep Ensembles are Bayesian
• OadTR: Online Action Detection with Transformers
• ViViT: A Video Vision Transformer
• Deep Deterministic Uncertainty: A Simple Baseline
• A Brief Review of Domain Adaptation
• Attentional Feature Fusion
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Artificial Intelligence, Values, and Alignment
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• Measuring Calibration in Deep Learning
• nuScenes: A multimodal dataset for autonomous driving
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Why ReLU networks yield high-confidence predictions far away from the\n training data and how to mitigate the problem
• Temporal Recurrent Networks for Online Action Detection
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• Understanding Measures of Uncertainty for Adversarial Example Detection
• Predictive Uncertainty Estimation via Prior Networks
• On Calibration of Modern Neural Networks
• Gradient Estimators for Implicit Models
• 1 year, 1000 km: The Oxford RobotCar dataset
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Variational Inference: A Review for Statisticians
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Scalable Bayesian Optimization Using Deep Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Stochastic Variational Inference
• The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
• Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
• Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'

Related

• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• Multidimensional Uncertainty Quantification for Deep Neural Networks
• Robust Deep Learning for Autonomous Driving
• A Survey on Uncertainty Quantification Methods for Deep Learning
• Uncertainty Quantification and Resource-Demanding Computer Vision Applications of Deep Learning