Delving into Out-of-Distribution Detection with Vision-Language Representations

2022/11/24 by Yifei Ming, Ming, Yifei, Ziyang Cai +9 · 30 citations
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2211.13445

openalex publication_date 2022/11/24 · openalex created_date 2022/11/30 · openalex updated_date 2026/07/28

Abstract

Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single modality (e.g., either vision or language), leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of OOD detection from a single-modal to a multi-modal regime. Particularly, we propose Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features with textual concepts. We contribute in-depth analysis and theoretical insights to understand the effectiveness of MCM. Extensive experiments demonstrate that MCM achieves superior performance on a wide variety of real-world tasks. MCM with vision-language features outperforms a common baseline with pure visual features on a hard OOD task with semantically similar classes by 13.1% (AUROC). Code is available at https://github.com/deeplearning-wisc/MCM.

Cited by

• Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring
• ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora
• Level, Sharpness, and Corpus: Why Zero-Shot OOD Detector Rankings Do Not Transfer
• LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks for Multimodal Large Language Models
• LOC: A General Language-Guided Framework for Open-Set 3D Occupancy Prediction
• Revisiting Logit Distributions for Reliable Out-of-Distribution Detection
• ΔEnergy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization
• Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detection
• Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection
• Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
• ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research
• Training-Free Label Space Alignment for Universal Domain Adaptation
• An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
• Prompt Optimization Meets Subspace Representation Learning for Few-shot Out-of-Distribution Detection
• ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoning
• PointAD+: Learning Hierarchical Representations for Zero-shot 3D Anomaly Detection
• RS-OOD: A Vision-Language Augmented Framework for Out-of-Distribution Detection in Remote Sensing
• M3OOD: Automatic Selection of Multimodal OOD Detectors
• Retrieval-Augmented Prompt for OOD Detection
• OpenHAIV: A Framework Towards Practical Open-World Learning
• Generalized Few-Shot Out-of-Distribution Detection
• COOkeD: Ensemble-based OOD detection in the era of zero-shot CLIP
• Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detection
• CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
• Synthesizing Near-Boundary OOD Samples for Out-of-Distribution Detection
• NegRefine: Refining Negative Label-Based Zero-Shot OOD Detection
• ViLU: Learning Vision-Language Uncertainties for Failure Prediction
• FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
• Spurious-Aware Prototype Refinement for Reliable Out-of-Distribution Detection
• Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagation

Related

• TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learning
• Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation
• Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection
• GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detection
• SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Models