The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Quality

2026/04/01 by Valery Manokhin · 1 voice
Computer Science · Mathematics · #Benchmark (surveying) #Binary number #Brier score #Calibration #Classifier (UML) #Explainable Artificial Intelligence (XAI) #Imbalanced Data Classification Techniques #Machine Learning and Data Classification #Pattern recognition (psychology) #Robustness (evolution) #Scoring rule #Suite #cs.LG #stat.ML

paper · pdf · doi:10.5281/zenodo.19372589

openalex publication_date 2026/04/01 · openalex created_date 2026/04/02 · arxiv published 2026/05/05 · arxiv updated 2026/05/05 · openalex updated_date 2026/08/01

Abstract

We introduce the Manokhin Probability Matrix, a two-dimensional diagnostic framework that classifies machine learning classifiers into four archetypes — Eagle (Type I), Bull (Type II), Sloth (Type III), and Mole (Type IV) — based on their calibration (Spiegelhalter Z-statistic) and discrimination (AUC-ROC) properties. The framework separates two quantities that the Brier score conflates: reliability (calibration error) and resolution (discriminatory ability). Using a large-scale empirical study of 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 benchmark suite (150 dataset-fold combinations), we assign each classifier to a quadrant and derive actionable prescriptions: - Eagles (CatBoost, TabICL, EBM, TabPFN, GBC, RF): well-calibrated, strong discriminators — ship as-is- Bulls (XGBoost, LightGBM, HGB, NCA, TabM): strong discriminators, poorly calibrated — fix with Venn-Abers- Sloths (LDA, TabTransformer, LR, SVM, AVG): well-calibrated but weak discriminators — retrain- Moles (ExtraTrees, RealMLP, NB, MLP, KNN): poor on both axes — start over Key contributions: (1) a formal proof that monotone calibrators cannot improve AUC (Proposition 1); (2) robustness analysis showing quadrant assignments are stable under absolute |Z|=1.96 thresholds; (3) empirical verification that AUC and Brier-resolution rankings yield 90.5% quadrant agreement (Spearman rho=0.948); (4) head-to-head Venn-Abers vs beta calibration comparison across all Bull models. Code, data, and experimental logs: https://github.com/valeman/classifier_calibration

Citations

• Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions
• Evaluating model calibration in classification
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• On Calibration of Modern Neural Networks
• Reliability, Sufficiency, and the Decomposition of Proper Scores
• Predicting good probabilities with supervised learning
• Applied Logistic Regression
• A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
• A New Vector Partition of the Probability Score
• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

Discussions

• https://bsky.app/profile/predict-addict.bsky.social/post/3mm4zxxfuur2o [bsky, 0 points, 0 comments]

Related

• Evaluating Probabilistic Classifiers: The Triptych
• Binary Classifier Calibration: Non-parametric approach
• Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods
• What Does It Take to Build a Performant Selective Classifier?
• Class-wise and reduced calibration methods