Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
2019/10/28 by Meelis Kull, Miquel Perelló-Nieto, Kull, Meelis +11 · 164 citations
Computer Science · Mathematics · #Algorithm #Artificial intelligence #Artificial neural network #Binary number #Brier score #Calibration #Computer science #Dirichlet distribution #Explainable Artificial Intelligence (XAI) #Extrapolation #Machine Learning and Data Classification #Mathematics #Multiplicative function #Neural Networks and Applications #Pattern recognition (psychology) #Probabilistic logic #Range (aeronautics) #Scaling #Softmax function #Statistics #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1910.12656
published in arXiv (Cornell University) (Cornell University) · Accepted for presentation at NeurIPS 2019
openalex created_date 2019/09/05 · arxiv created 2019/10/28 · openalex publication_date 2019/10/28 · arxiv updated 2019/10/29 · openalex updated_date 2026/07/28
Abstract
Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.
Cited by
- Brenier Isotonic Regression
- Gradient-Based Latent Decomposition Reveals Mechanisms of Feature Degradation in Weakly Supervised Mammography
- A Diagnostic Gap Framework for Evaluating Reconstruction Fidelity in Weakly Supervised Mammography
- Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
- Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques
- Graph Memory: A Structured and Interpretable Framework for Modality-Agnostic Embedding-Based Inference
- Edge-aware baselines for ogbn-proteins in PyTorch Geometric: species-wise normalization, post-hoc calibration, and cost-accuracy trade-offs
- Reliability Assessment Framework Based on Feature Separability for Pathological Cell Image Classification under Prior Bias
- ActiveGrasp: Information-Guided Active Grasping with Calibrated Energy-based Model
- Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
- Uncertainty Calibration of Multi-Label Bird Sound Classifiers
- Calibrating Deep Neural Networks using Focal Loss
- Structured Matrix Scaling for Multi-Class Calibration
- A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysis
- Multiclass Local Calibration With the Jensen-Shannon Distance
- Scalable Utility-Aware Multiclass Calibration
- Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
- Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty
- Soft Calibration Objectives for Neural Networks
- Navigating prevalence shifts in image analysis algorithm deployment
- Probabilistic Object Classification using CNN ML-MAP layers
- Machine Learning based Analysis for Radiomics Features Robustness in Real-World Deployment Scenarios
- Robust Decision Making with Partially Calibrated Forecasts
- What Does It Take to Build a Performant Selective Classifier?
- Calibration and Discrimination Optimization Using Clusters of Learned Representation
- Uncertainty evaluation of segmentation models for Earth observation
- Confidence Calibration with Bounded Error Using Transformations
- CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning
- Fast and Scalable Score-Based Kernel Calibration Tests
- UNCAP: Uncertainty-Guided Planning Using Natural Language Communication for Cooperative Autonomous Vehicles
- Private Federated Multiclass Post-hoc Calibration
- Calibrating the Full Predictive Class Distribution of 3D Object Detectors for Autonomous Driving
- Truthful Calibration Errors for Multi-Class Prediction
- Calibration Meets Reality: Making Machine Learning Predictions Trustworthy
- Confidence-Calibrating Regularization for Robust Brain MRI Segmentation Under Domain Shift
- Sample-efficient Multiclass Calibration under ℓp Error
- Objective-Aligned Direct Answer SFT for Robust Multi-Frame Medical VQA
- Towards Trustworthy Breast Tumor Segmentation in Ultrasound using Monte Carlo Dropout and Deep Ensembles for Epistemic Uncertainty Estimation
- Similarity-Distance-Magnitude Activations
- Top-label calibration and multiclass-to-binary reductions
- Pseudo-D: Informing Multi-View Uncertainty Estimation with Calibrated Neural Training Dynamics
- Composable Score-based Graph Diffusion Model for Multi-Conditional Molecular Generation
- Split Conformal Prediction in the Function Space with Neural Operators
- Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization
- Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal
- Gradient Rectification for Robust Calibration under Distribution Shift
- SNAP-UQ: Self-supervised Next-Activation Prediction for Single-Pass Uncertainty in TinyML
- PEdger++: Practical Edge Detection via Assembling Cross Information
- Grasp-HGN: Grasping the Unexpected
- Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis
- Uncertainty-aware Predict-Then-Optimize Framework for Equitable Post-Disaster Power Restoration
- Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring
- Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modeling
- Semantic-Aware Gaussian Process Calibration with Structured Layerwise Kernels for Deep Neural Networks
- On Deep Neural Network Calibration by Regularization and its Impact on Refinement
- Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
- Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
- Temporal Probability Calibration
- MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator
- To Trust or Not to Trust: On Calibration in ML-based Resource Allocation for Wireless Networks
- Calibrating Deep Neural Network Classifiers on Out-of-Distribution Datasets
- Leveraging Beam Search Information for Confidence Estimation in E2E ASR
- Instance-Wise Monotonic Calibration by Constrained Transformation
- Disentangling Label Distribution for Long-tailed Visual Recognition
- Sample Margin-Aware Recalibration of Temperature Scaling
- Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classification
- Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
- Towards Reliable Detection of Empty Space: Conditional Marked Point Processes for Object Detection
- h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective
- Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmax
- Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness
- Improving model calibration with accuracy versus uncertainty optimization
- Class-Similarity Based Label Smoothing for Confidence Calibration
- Distribution-free uncertainty quantification for classification under label shift
- Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models
- Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
- Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Loss
- Practical estimation of the optimal classification error with soft labels and calibration
- Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequencies
- Identifying and Exploiting Structures for Reliable Deep Learning
- Uniform convergence of the smooth calibration error and its relationship with functional gradient
- Improving Classifier Confidence using Lossy Label-Invariant Transformations
- C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models
- Calibration of Neural Networks using Splines
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator
- Adaptive Cumulative Mass Calibration with Conformal Prediction
- Quantile Regularization: Towards Implicit Calibration of Regression Models
- Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
- Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)
- The Final Layer Holds the Key: A Unified and Efficient GNN Calibration Framework
- On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift
- Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models
- Nonparametric Distribution Regression Re-calibration
- Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
- Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
- Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review
- Three Types of Calibration with Properties and their Semantic and Formal Relationships
- Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems
- Beyond One-Hot Labels: Semantic Mixing for Model Calibration
- Evidential Rule Learning for Interpretable Classification with Abstention
- Towards Calibration Enhanced Network by Inverse Adversarial Attack
Related