Vision Foundation Model Embedding-Based Semantic Anomaly Detection

2025/05/12 by Ronecker, Max Peter, Foutter, Matthew, Elhafsi, Amine +4 · 2 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2505.07998

Abstract

Semantic anomalies are contextually invalid or unusual combinations of familiar visual elements that can cause undefined behavior and failures in system-level reasoning for autonomous systems. This work explores semantic anomaly detection by leveraging the semantic priors of state-of-the-art vision foundation models, operating directly on the image. We propose a framework that compares local vision embeddings from runtime images to a database of nominal scenarios in which the autonomous system is deemed safe and performant. In this work, we consider two variants of the proposed framework: one using raw grid-based embeddings, and another leveraging instance segmentation for object-centric representations. To further improve robustness, we introduce a simple filtering mechanism to suppress false positives. Our evaluations on CARLA-simulated anomalies show that the instance-based method with filtering achieves performance comparable to GPT-4o, while providing precise anomaly localization. These results highlight the potential utility of vision embeddings from foundation models for real-time anomaly detection in autonomous systems.

Citations

• SAM 2: Segment Anything in Images and Videos
• Segment Every Out-of-Distribution Object
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Vision Transformers Need Registers
• Scaling Open-Vocabulary Object Detection
• Semantic Anomaly Detection with Large Language Models
• DINOv2: Learning Robust Visual Features without Supervision
• A System-Level View on Out-of-Distribution Data in Robotics
• Masked Autoencoders Are Scalable Vision Learners
• On the Opportunities and Risks of Foundation Models
• Emerging Properties in Self-Supervised Vision Transformers
• Learning Transferable Visual Models From Natural Language Supervision

Cited by

• NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving
• Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis
• Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety

Related

• Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning
• Semantic Anomaly Detection with Large Language Models
• Pixel-wise Anomaly Detection in Complex Driving Scenes
• AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models
• Anomaly Detection by Adapting a pre-trained Vision Language Model