2017/07/25 by Shujian Yu, Zubin Abraham, Yu, Shujian +9 · 1 citation
Computer Science · Mathematics · #Anomaly Detection Techniques and Applications #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Network Security and Intrusion Detection #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1707.07821
Manuscript accepted by the Journal of The Franklin Institute. A short version of this manuscript, titled "Concept Drift Detection with Hierarchical Hypothesis Test", was presented at the 2017 SIAM International Conference on Data Mining (SDM) https://epubs.siam.org/doi/10.1137/1.9781611974973.86
openalex publication_date 2017/07/25 · openalex created_date 2017/07/31 · arxiv created 2019/02/08 · arxiv updated 2019/02/11 · openalex updated_date 2026/07/28
A fundamental issue for statistical classification models in a streaming environment is that the joint distribution between predictor and response variables changes over time (a phenomenon also known as concept drifts), such that their classification performance deteriorates dramatically. In this paper, we first present a hierarchical hypothesis testing (HHT) framework that can detect and also adapt to various concept drift types (e.g., recurrent or irregular, gradual or abrupt), even in the presence of imbalanced data labels. A novel concept drift detector, namely Hierarchical Linear Four Rates (HLFR), is implemented under the HHT framework thereafter. By substituting a widely-acknowledged retraining scheme with an adaptive training strategy, we further demonstrate that the concept drift adaptation capability of HLFR can be significantly boosted. The theoretical analysis on the Type-I and Type-II errors of HLFR is also performed. Experiments on both simulated and real-world datasets illustrate that our methods outperform state-of-the-art methods in terms of detection precision, detection delay as well as the adaptability across different concept drift types.