Post-hoc Uncertainty Calibration for Domain Drift Scenarios

2020/12/20 by Christian Tomani, Sebastian Gruber, Tomani, Christian +7 · 8 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2012.10988

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Code available at https://github.com/tochris/calibration-domain-drift

arxiv created 2021/06/23 · arxiv updated 2021/06/24

Abstract

We address the problem of uncertainty calibration. While standard deep neural networks typically yield uncalibrated predictions, calibrated confidence scores that are representative of the true likelihood of a prediction can be achieved using post-hoc calibration methods. However, to date the focus of these approaches has been on in-domain calibration. Our contribution is two-fold. First, we show that existing post-hoc calibration methods yield highly over-confident predictions under domain shift. Second, we introduce a simple strategy where perturbations are applied to samples in the validation set before performing the post-hoc calibration step. In extensive experiments, we demonstrate that this perturbation step results in substantially better calibration under domain shift on a wide range of architectures and modelling tasks.

Cited by

• Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning
• Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration
• LoFT: Parameter-Efficient Fine-Tuning for Long-tailed Semi-Supervised Learning in Open-World Scenarios
• Gradient Rectification for Robust Calibration under Distribution Shift
• CaliMatch: Adaptive Calibration for Improving Safe Semi-supervised Learning
• Where are we with calibration under dataset shift in image classification?
• When Small Guides Large: Cross-Model Co-Learning for Test-Time Adaptation
• Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanations

Related

• Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration
• Beyond In-Domain Scenarios: Robust Density-Aware Calibration
• Uncertainty-Aware Post-Hoc Calibration: Mitigating Confidently Incorrect Predictions Beyond Calibration Metrics
• Improving model calibration with accuracy versus uncertainty optimization
• Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors