A Variational Information Theoretic Approach to Out-of-Distribution Detection

2025/06/17 by Sudeepta Mondal, Mondal, Sudeepta, Zhuolin Jiang +3 · 2 citations
Computer Science · Decision Sciences · Engineering · #Advanced Statistical Process Monitoring #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.14194

openalex publication_date 2025/06/17 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

We present a theory for the construction of out-of-distribution (OOD) detection features for neural networks. We introduce random features for OOD through a novel information-theoretic loss functional consisting of two terms, the first based on the KL divergence separates resulting in-distribution (ID) and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information. We formulate a variational procedure to optimize the loss and obtain OOD features. Based on assumptions on OOD distributions, one can recover properties of existing OOD features, i.e., shaping functions. Furthermore, we show that our theory can predict a new shaping function that out-performs existing ones on OOD benchmarks. Our theory provides a general framework for constructing a variety of new features with clear explainability.

Citations

• Towards Optimal Feature-Shaping Methods for Out-of-Distribution Detection
• LINe: Out-of-Distribution Detection by Leveraging Important Neurons
• VRA: Variational Rectified Activation for Out-of-distribution Detection
• Energy-based Out-of-Distribution Detection for Graph Neural Networks
• Extremely Simple Activation Shaping for Out-of-Distribution Detection
• Out-of-Distribution Detection with Deep Nearest Neighbors
• ViM: Out-Of-Distribution with Virtual-logit Matching
• ReAct: Out-of-distribution Detection With Rectified Activations
• On Out-of-distribution Detection with Energy-based Models
• Exploring the Limits of Out-of-Distribution Detection
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Energy-based Out-of-distribution Detection
• Why Normalizing Flows Fail to Detect Out-of-Distribution Data
• Natural Adversarial Examples
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• A Tutorial on Bayesian Optimization
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Identity Mappings in Deep Residual Networks
• TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
• Describing Textures in the Wild
• The information bottleneck method
• Scaling Out-of-Distribution Detection for Real-World Settings

Cited by

• Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction
• Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking

Related

• Provable Guarantees for Understanding Out-of-distribution Detection
• Towards Optimal Feature-Shaping Methods for Out-of-Distribution Detection
• Out-of-Distribution Detection Using Outlier Detection Methods
• Gradient-Regularized Out-of-Distribution Detection
• Why Should we Combine Training and Post-Training Methods for Out-of-Distribution Detection?