2020/01/01 by Theodore James Thibault Heiser, Mari-Liis Allikivi, Meelis Kull
Computer Science · #Algorithm #Anomaly Detection Techniques and Applications #Artificial intelligence #Brier score #Class (philosophy) #Classifier (UML) #Computer science #Cross entropy #Data mining #Entropy (arrow of time) #Imbalanced Data Classification Techniques #Machine Learning and Data Classification #Machine learning #Pattern recognition (psychology) #Principle of maximum entropy #Probabilistic logic #Scoring rule #cs.LG
paper · pdf · doi:10.1007/978-3-030-46147-8_4
published as ECML PKDD 2019. Lecture Notes in Computer Science, vol 11907. Springer, Cham (2020) · ECML PKDD 2019 conference paper, 16 pages
openalex publication_date 2020/01/01 · arxiv created 2021/11/03 · arxiv updated 2021/11/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Minimizing expected loss measured by a proper scoring rule, such as Brier score or log-loss (cross-entropy), is a common objective while training a probabilistic classifier. If the data have experienced dataset shift where the class distributions change post-training, then often the model's performance will decrease, over-estimating the probabilities of some classes while under-estimating the others on average. We propose unbounded and bounded general adjustment (UGA and BGA) methods that transform all predictions to (re-)equalize the average prediction and the class distribution. These methods act differently depending on which proper scoring rule is to be minimized, and we have a theoretical guarantee of reducing loss on test data, if the exact class distribution is known. We also demonstrate experimentally that, when in practice the class distribution is known only approximately, there is often still a reduction in loss depending on the amount of shift and the precision to which the class distribution is known.