An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity

2025/09/16 by Y. S. Lee, Lee, Yuxiao, Wei Ye +5
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2509.13375

openalex publication_date 2025/09/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable zero-shot out-of-distribution (OOD) detection capabilities, vital for reliable AI systems. Despite this promising capability, a comprehensive understanding of (1) why they work so effectively, (2) what advantages do they have over single-modal methods, and (3) how is their behavioral robustness -- remains notably incomplete within the research community. This paper presents a systematic empirical analysis of VLM-based OOD detection using in-distribution (ID) and OOD prompts. (1) Mechanisms: We systematically characterize and formalize key operational properties within the VLM embedding space that facilitate zero-shot OOD detection. (2) Advantages: We empirically quantify the superiority of these models over established single-modal approaches, attributing this distinct advantage to the VLM's capacity to leverage rich semantic novelty. (3) Sensitivity: We uncovers a significant and previously under-explored asymmetry in their robustness profile: while exhibiting resilience to common image noise, these VLM-based methods are highly sensitive to prompt phrasing. Our findings contribute a more structured understanding of the strengths and critical vulnerabilities inherent in VLM-based OOD detection, offering crucial, empirically-grounded guidance for developing more robust and reliable future designs.

Citations

• Learning Transferable Negative Prompts for Out-of-Distribution Detection
• Negative Label Guided OOD Detection with Pretrained Vision-Language Models
• Learning with Mixture of Prototypes for Out-of-Distribution Detection
• NECO: NEural Collapse Based Out-of-distribution detection
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• Fine-Grained Visual Prompting
• LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning
• On Evaluating Adversarial Robustness of Large Vision-Language Models
• Micrograph segmentations for DDEVD
• Segment Anything
• A Survey of Large Language Models
• Sigmoid Loss for Language Image Pre-Training
• Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Robust Speech Recognition via Large-Scale Weak Supervision
• Delving into Out-of-Distribution Detection with Vision-Language Representations
• Magic3D: High-Resolution Text-to-3D Content Creation
• EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
• Is Out-of-Distribution Detection Learnable?
• OpenOOD: Benchmarking Generalized Out-of-Distribution Detection
• Diffusion Models: A Comprehensive Survey of Methods and Applications
• Out-of-Distribution Detection with Semantic Mismatch under Masking
• Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities
• CyCLIP: Cyclic Contrastive Language-Image Pretraining
• How Useful are Gradients for OOD Detection Really?
• VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
• Out-of-Distribution Detection with Deep Nearest Neighbors
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Conditional Prompt Learning for Vision-Language Models
• VOS: Learning What You Don't Know by Virtual Outlier Synthesis
• CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields
• Provable Guarantees for Understanding Out-of-distribution Detection
• DICE: Leveraging Sparsification for Out-of-Distribution Detection
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Generalized Out-of-Distribution Detection: A Survey
• Generalized Out-of-Distribution Detection: A Survey
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• On the Importance of Gradients for Detecting Distributional Shifts in the Wild
• ActionCLIP: A New Paradigm for Video Action Recognition
• Towards Out-Of-Distribution Generalization: A Survey
• Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
• Exploring the Limits of Out-of-Distribution Detection
• MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Energy-based Out-of-distribution Detection
• Cross-Sectional Studies
• Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Why ReLU networks yield high-confidence predictions far away from the\n training data and how to mitigate the problem
• Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• Adversarially Learned One-Class Classifier for Novelty Detection
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Deep Residual Learning for Image Recognition
• Towards Open Set Deep Networks
• Describing Textures in the Wild
• GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detection
• Scaling Out-of-Distribution Detection for Real-World Settings

Related

• Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection
• VisTa: Visual-contextual and Text-augmented Zero-shot Object-level OOD Detection
• Negative Label Guided OOD Detection with Pretrained Vision-Language Models
• Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Survey
• SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Models