Statistical Inference for Clustering-based Anomaly Detection

2025/04/25 by Nguyen Thi Minh Phu, Phu, Nguyen Thi Minh, Duong Tan Loc +3
Computer Science · #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Network Security and Intrusion Detection #Software System Performance and Reliability

paper · pdf · doi:10.48550/arxiv.2504.18633

openalex publication_date 2025/04/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Unsupervised anomaly detection (AD) is a fundamental problem in machine learning and statistics. A popular approach to unsupervised AD is clustering-based detection. However, this method lacks the ability to guarantee the reliability of the detected anomalies. In this paper, we propose SI-CLAD (Statistical Inference for CLustering-based Anomaly Detection), a novel statistical framework for testing the clustering-based AD results. The key strength of SI-CLAD lies in its ability to rigorously control the probability of falsely identifying anomalies, maintaining it below a pre-specified significance level α (e.g., α= 0.05). By analyzing the selection mechanism inherent in clustering-based AD and leveraging the Selective Inference (SI) framework, we prove that false detection control is attainable. Moreover, we introduce a strategy to boost the true detection rate, enhancing the overall performance of SI-CLAD. Extensive experiments on synthetic and real-world datasets provide strong empirical support for our theoretical findings, showcasing the superior performance of the proposed method.

Citations

• Quantifying Statistical Significance of Deep Nearest Neighbor Anomaly Detection via Selective Inference
• Controllable RANSAC-based Anomaly Detection via Hypothesis Testing
• Statistical Test for Attention Map in Vision Transformer
• Valid P-Value for Deep Learning-Driven Salient Region
• Valid and Exact Statistical Inference for Multi-dimensional Multiple Change-Points by Selective Inference
• Exact Statistical Inference for the Wasserstein Distance by Selective\n Inference
• Tree-Values: selective inference for regression trees
• Fast and More Powerful Selective Inference for Sparse High-order Interaction Model
• Conditional Selective Inference for Robust Regression and Outlier Detection using Piecewise-Linear Homotopy Continuation
• Clustering-based Anomaly Detection in Multivariate Time Series Data
• Selective Inference for Hierarchical Clustering
• Quantifying Statistical Significance of Neural Network-based Image Segmentation by Selective Inference
• Computing Valid p-value for Optimal Changepoint by Selective Inference using Dynamic Programming
• Testing for a Change in Mean After Changepoint Detection
• Computing Valid p-values for Image Segmentation by Selective Inference
• Post-Selection Inference for Changepoint Detection Algorithms with Application to Copy Number Variation Data
• More powerful post-selection inference, with application to the Lasso
• Valid Inference Corrected for Outlier Removal
• Selective Inference for Change Point Detection in Multi-dimensional Sequences
• Post Selection Inference with Kernels
• Selective Sequential Model Selection
• Optimal Inference After Model Selection
• A significance test for forward stepwise model selection
• Exact Post-Selection Inference for Sequential Regression Procedures
• Exact post-selection inference, with application to the lasso
• Valid post-selection inference
• Efficient algorithms for mining outliers from large data sets

Related

• Towards a Unified Framework of Clustering-based Anomaly Detection
• AnoOnly: Semi-Supervised Anomaly Detection with the Only Loss on Anomalies
• When Unsupervised Domain Adaptation meets One-class Anomaly Detection: Addressing the Two-fold Unsupervised Curse by Leveraging Anomaly Scarcity
• Unsupervised Surrogate Anomaly Detection
• A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithms