Automating concept-drift detection by self-evaluating predictive model degradation

2019/07/18 by Cerquitelli, Tania, Proto, Stefano, Ventura, Francesco +2
#FOS: Computer and information sciences #I.2 #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1907.08120

Abstract

A key aspect of automating predictive machine learning entails the capability of properly triggering the update of the trained model. To this aim, suitable automatic solutions to self-assess the prediction quality and the data distribution drift between the original training set and the new data have to be devised. In this paper, we propose a novel methodology to automatically detect prediction-quality degradation of machine learning models due to class-based concept drift, i.e., when new data contains samples that do not fit the set of class labels known by the currently-trained predictive model. Experiments on synthetic and real-world public datasets show the effectiveness of the proposed methodology in automatically detecting and describing concept drift caused by changes in the class-label data distributions.

Related

• From concept drift to model degradation: An overview on performance-aware drift detectors
• From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectors
• Flexible and Efficient Drift Detection without Labels
• Time to Retrain? Detecting Concept Drifts in Machine Learning Systems
• On the Reliable Detection of Concept Drift from Streaming Unlabeled Data