Hierarchical VAEs Know What They Don't Know

2021/02/16 by Jakob D. Havtorn, Havtorn, Jakob D., Jes Frellsen +5 · 8 citations
Computer Science · Mathematics · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Artificial intelligence #Benchmark (surveying) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Context (archaeology) #Data mining #Data set #Estimator #FOS: Computer and information sciences #Feature (linguistics) #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Generative grammar #Generative model #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematics #Set (abstract data type) #Statistics #cs.AI #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2102.08248

published in arXiv (Cornell University), 4117-4128 (Cornell University) · Appeared in Proceedings of the 38th International Conference on Machine Learning (ICML 2021). 18 pages, source code available at https://github.com/JakobHavtorn/hvae-oodd, https://github.com/vlievin/biva-pytorch and https://github.com/larsmaaloee/BIVA

openalex publication_date 2021/02/16 · arxiv created 2022/01/18 · arxiv updated 2022/01/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

Deep generative models have been demonstrated as state-of-the-art density estimators. Yet, recent work has found that they often assign a higher likelihood to data from outside the training distribution. This seemingly paradoxical behavior has caused concerns over the quality of the attained density estimates. In the context of hierarchical variational autoencoders, we provide evidence to explain this behavior by out-of-distribution data having in-distribution low-level features. We argue that this is both expected and desirable behavior. With this insight in hand, we develop a fast, scalable and fully unsupervised likelihood-ratio score for OOD detection that requires data to be in-distribution across all feature-levels. We benchmark the method on a vast set of data and model combinations and achieve state-of-the-art results on out-of-distribution detection.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Gradient-based learning applied to document recognition
• Representation Learning: A Review and New Perspectives
• Human-level concept learning through probabilistic program induction
• Pixel Recurrent Neural Networks
• Auxiliary Deep Generative Models
• WaveNet: A Generative Model for Raw Audio
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Inference Suboptimality in Variational Autoencoders
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Avoiding Latent Variable Collapse With Generative Skip Models
• Deep Anomaly Detection with Outlier Exposure
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Learning Multiple Layers of Features from Tiny Images

Cited by

• DecNefSimulator: A Modular, Interpretable Framework for Decoded Neurofeedback Simulation Using Generative Models
• Evolutionary Computation as Natural Generative AI
• Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs
• Optimizing Latent Dimension Allocation in Hierarchical VAEs: Balancing Attenuation and Information Retention for OOD Detection
• Robust Out-of-Distribution Detection on Deep Probabilistic Generative Models
• Exploring bidirectional bounds for minimax-training of Energy-based models
• VaCDA: Variational Contrastive Alignment-based Scalable Human Activity Recognition

Related

• Unsupervised Out-of-Distribution Detection by Restoring Lossy Inputs with Variational Autoencoder
• OOD Detection with immature Models
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• Understanding Failures in Out-of-Distribution Detection with Deep Generative Models
• Probability Density from Latent Diffusion Models for Out-of-Distribution Detection