Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes

2021/11/24 by Yu Tian, Yuyuan Liu, Tian, Yu +9 · 19 citations
Computer Science · Engineering · Mathematics · #Anomaly (physics) #Anomaly Detection Techniques and Applications #Anomaly detection #Artificial intelligence #Autonomous Vehicle Technology and Safety #Code (set theory) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Energy (signal processing) #FOS: Computer and information sciences #Mathematics #Outlier #Pattern recognition (psychology) #Pixel #Segmentation #Statistics #Video Surveillance and Tracking Methods #cs.CV

paper · pdf · doi:10.48550/arxiv.2111.12264

published in arXiv (Cornell University) (Cornell University) · ECCV 2022 Oral

openalex publication_date 2021/11/24 · arxiv created 2022/09/05 · arxiv updated 2022/09/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

State-of-the-art (SOTA) anomaly segmentation approaches on complex urban driving scenes explore pixel-wise classification uncertainty learned from outlier exposure, or external reconstruction models. However, previous uncertainty approaches that directly associate high uncertainty to anomaly may sometimes lead to incorrect anomaly predictions, and external reconstruction models tend to be too inefficient for real-time self-driving embedded systems. In this paper, we propose a new anomaly segmentation method, named pixel-wise energy-biased abstention learning (PEBAL), that explores pixel-wise abstention learning (AL) with a model that learns an adaptive pixel-level anomaly class, and an energy-based model (EBM) that learns inlier pixel distribution. More specifically, PEBAL is based on a non-trivial joint training of EBM and AL, where EBM is trained to output high-energy for anomaly pixels (from outlier exposure) and AL is trained such that these high-energy pixels receive adaptive low penalty for being included to the anomaly class. We extensively evaluate PEBAL against the SOTA and show that it achieves the best performance across four benchmarks. Code is available at https://github.com/tianyu0207/PEBAL.

Citations

• The Pascal Visual Object Classes (VOC) Challenge
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Predictive Uncertainty Estimation via Prior Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Discriminative out-of-distribution detection for semantic segmentation
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Deep Anomaly Detection with Outlier Exposure
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity
• Scaling Out-of-Distribution Detection for Real-World Settings

Cited by

• ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Data
• SupLID: Geometrical Guidance for Out-of-Distribution Detection in Semantic Segmentation
• Relative Energy Learning for LiDAR Out-of-Distribution Detection
• Leveraging Text-Driven Semantic Variation for Robust OOD Segmentation
• Training-Free Out-Of-Distribution Segmentation With Foundation Models
• Road Obstacle Video Segmentation
• From Pixel to Mask: A Survey of Out-of-Distribution Segmentation
• Uncertainty-Aware Likelihood Ratio Estimation for Pixel-Wise Out-of-Distribution Detection
• Distributional Uncertainty for Out-of-Distribution Detection
• CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning
• OoDDINO:A Multi-level Framework for Anomaly Segmentation on Complex Road Scenes
• Out-of-Distribution Semantic Occupancy Prediction
• Panoramic Out-of-Distribution Segmentation
• Anomalies by Synthesis: Anomaly Detection using Generative Diffusion Models for Off-Road Navigation
• A2Seek: Towards Reasoning-Centric Benchmark for Aerial Anomaly Understanding
• Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation
• Spotting the Unexpected (STU): A 3D LiDAR Dataset for Anomaly Segmentation in Autonomous Driving
• Open-set Anomaly Segmentation in Complex Scenarios
• Prior2Former -- Evidential Modeling of Mask Transformers for Assumption-Free Open-World Panoptic Segmentation

Related

• Self-Supervised Likelihood Estimation with Energy Guidance for Anomaly Segmentation in Urban Scenes
• Pixel-wise Anomaly Detection in Complex Driving Scenes
• Random Walk on Pixel Manifolds for Anomaly Segmentation of Complex Driving Scenes
• UGainS: Uncertainty Guided Anomaly Instance Segmentation
• OoDDINO:A Multi-level Framework for Anomaly Segmentation on Complex Road Scenes