Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features

2020/06/18 by Robin Tibor Schirrmeister, Schirrmeister, Robin Tibor, Yuxuan Zhou +5 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Network Security and Intrusion Detection #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.2006.10848

openalex publication_date 2020/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep generative networks trained via maximum likelihood on a natural image dataset like CIFAR10 often assign high likelihoods to images from datasets with different objects (e.g., SVHN). We refine previous investigations of this failure at anomaly detection for invertible generative networks and provide a clear explanation of it as a combination of model bias and domain prior: Convolutional networks learn similar low-level feature distributions when trained on any natural image dataset and these low-level features dominate the likelihood. Hence, when the discriminative features between inliers and outliers are on a high-level, e.g., object shapes, anomaly detection becomes particularly challenging. To remove the negative impact of model bias and domain prior on detecting high-level differences, we propose two methods, first, using the log likelihood ratios of two identical models, one trained on the in-distribution data (e.g., CIFAR10) and the other one on a more general distribution of images (e.g., 80 Million Tiny Images). We also derive a novel outlier loss for the in-distribution network on samples from the more general distribution to further improve the performance. Secondly, using a multi-scale model like Glow, we show that low-level features are mainly captured at early scales. Therefore, using only the likelihood contribution of the final scale performs remarkably well for detecting high-level feature differences of the out-of-distribution and the in-distribution. This method is especially useful if one does not have access to a suitable general distribution. Overall, our methods achieve strong anomaly detection performance in the unsupervised setting, and only slightly underperform state-of-the-art classifier-based methods in the supervised setting. Code can be found at https://github.com/boschresearch/hierarchical_anomaly_detection.

Citations

• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Density Estimation Using Real NVP
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Deep Anomaly Detection with Outlier Exposure
• OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations
• Out-of-Distribution Detection Using Neural Rendering Generative Models
• Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• INPUT COMPLEXITY AND OUT-OF-DISTRIBUTION DETECTION WITH LIKELIHOOD-BASED GENERATIVE MODELS
• Classification-Based Anomaly Detection for General Data
• Contrastive Training for Improved Out-of-Distribution Detection
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Enhancing ECG Classification Robustness with Lightweight Unsupervised Anomaly Detection Filters
• OCSVM-Guided Representation Learning for Unsupervised Anomaly Detection

Related

• Deep Anomaly Detection with Outlier Exposure
• Deep Generative Model using Unregularized Score for Anomaly Detection with Heterogeneous Complexity
• GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training
• Unsupervised Learning of Anomaly Detection from Contaminated Image Data using Simultaneous Encoder Training
• Anomaly Detection Using GANs for Visual Inspection in Noisy Training Data