Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

2019/06/06 by Yaniv Ovadia, Ovadia, Yaniv, Emily Fertig +16 · 227 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Gaussian Processes and Bayesian Inference #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1906.02530

Advances in Neural Information Processing Systems, 2019

arxiv created 2019/12/17 · arxiv updated 2019/12/24

Abstract

Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive \em uncertainty. Quantifying uncertainty is especially critical in real-world settings, which often involve input distributions that are shifted from the training distribution due to a variety of factors including sample bias and non-stationarity. In such settings, well calibrated uncertainty estimates convey information about when a model's output should (or should not) be trusted. Many probabilistic deep learning methods, including Bayesian-and non-Bayesian methods, have been proposed in the literature for quantifying predictive uncertainty, but to our knowledge there has not previously been a rigorous large-scale empirical comparison of these methods under dataset shift. We present a large-scale benchmark of existing state-of-the-art methods on classification problems and investigate the effect of dataset shift on accuracy and calibration. We find that traditional post-hoc calibration does indeed fall short, as do several other previous methods. However, some methods that marginalize over models give surprisingly strong results across a broad spectrum of tasks.

Citations

• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
• Semi-Supervised Learning with Deep Generative Models
• Gradient-based learning applied to document recognition
• End to End Learning for Self-Driving Cars
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

Cited by

• Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
• Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation
• Bayesian Complete-Pooling in Cross-Subject Classification for Motor Imagery Electroencephalogram
• Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification
• Toward Ethical AI Through Bayesian Uncertainty in Neural Question Answering
• Mimicking Human Visual Development for Learning Robust Image Representations
• ProbeMDE: Uncertainty-Guided Active Proprioception for Monocular Depth Estimation in Surgical Robotics
• Interoceptive machine framework: Toward interoception-inspired regulatory architectures in artificial intelligence
• Calibrating Uncertainty for Zero-Shot Adversarial CLIP
• Uncertainty-Aware Subset Selection for Robust Visual Explainability under Distribution Shifts
• Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data
• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• How (Mis)calibrated is Your Federated CLIP and What To Do About It?
• SweetDeep: A Wearable AI Solution for Real-Time Non-Invasive Diabetes Screening
• Credal Graph Neural Networks
• CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regression
• Probabilistic Fusion and Calibration of Neural Speaker Diarization Models
• PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logic
• Black-Box Uncertainty Estimation for Deep Learning Models in Atomistic Simulations
• Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation
• Uncertainty-Guided Selective Adaptation Enables Cross-Platform Predictive Fluorescence Microscopy
• Bayesian--AI Fusion for Epidemiological Decision Making: Calibrated Risk, Honest Uncertainty, and Hyperparameter Intelligence
• A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts
• Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration
• Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification
• ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora
• Adaptive Sample-Level Framework Motivated by Distributionally Robust Optimization with Variance-Based Radius Assignment for Enhanced Neural Network Generalization Under Distribution Shift
• Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering
• Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features
• CalTwin: Towards Calibrated, Shift-Robust Medical World Models via Fisher-Information Regularisation
• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels
• Mitigating Bias in Calibration Error Estimation
• Optimality of Sub-network Laplace Approximations: New Results and Methods
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Confronting verbalized uncertainty: Understanding how LLM’s verbalized uncertainty influences users in AI-assisted decision-making
• Challenges in data-driven geospatial modeling for environmental research and practice
• Bayesian Attention Modules
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Robust learning from literature data: Model generalizability and uncertainty for predicting conjugated polymer solution conformation
• An uncertainty-aware physics-informed neural network solution for the Black-Scholes equation: a novel framework for option pricing
• Uncertainty-Aware Autonomous Vehicles: Predicting the Road Ahead
• Frequentist Validity of Epistemic Uncertainty Estimators
• Worst-Case Distance-Aware Error Bounds for Neural Networks
• Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Method
• On Uncertainty Calibration for Equivariant Functions
• What Does It Take to Build a Performant Selective Classifier?
• A Kernel Framework to Quantify a Model's Local Predictive Uncertainty under Data Distributional Shifts
• Uncertainty evaluation of segmentation models for Earth observation
• Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction
• Functional Distribution Networks (FDN)
• Black-box Optimization of LLM Outputs by Asking for Directions
• Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?
• Adaptive Individual Uncertainty under Out-Of-Distribution Shift with Expert-Routed Conformal Prediction
• Towards Adversarial Robustness and Uncertainty Quantification in DINOv2-based Few-Shot Anomaly Detection
• Federated Conditional Conformal Prediction via Generative Models
• Multivariate Deep Evidential Regression
• Self-Training with Dynamic Weighting for Robust Gradual Domain Adaptation
• A Machine Learning Perspective on Automated Driving Corner Cases
• Tight Robustness Certificates and Wasserstein Distributional Attacks for Deep Neural Networks
• Bi-level Meta-Policy Control for Dynamic Uncertainty Calibration in Evidential Deep Learning
• Towards Neurocognitive-Inspired Intelligence: From AI's Structural Mimicry to Human-Like Functional Cognition
• Scalable Offline Metrics for Autonomous Driving
• Scalable Bayesian neural networks by layer-wise input augmentation
• Conformalized Gaussian processes for online uncertainty quantification over graphs
• Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection
• Domain-Shift-Aware Conformal Prediction for Large Language Models
• Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking
• Uncertainty-Aware Deep Learning for Wildfire Danger Forecasting
• Generalized Correctness Models: Learning Calibrated and Model-Agnostic Correctness Predictors from Historical Patterns
• Calibration Meets Reality: Making Machine Learning Predictions Trustworthy
• Calibrated and Resource-Aware Super-Resolution for Reliable Driver Behavior Analysis
• The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models
• When Derived Measurements Mislead: Quantifying and Mitigating LLM Over-Trust with Privileged-Modality Reliability Evidence
• On-manifold Adversarial Data Augmentation Improves Uncertainty Calibration
• Identifying Adaptive Footprints in the Presence of Demographic Uncertainty
• When and How Mixup Improves Calibration
• Automated Labeling of Intracranial Arteries with Uncertainty Quantification Using Deep Learning
• Uncertainty-Supervised Interpretable and Robust Evidential Segmentation
• Calibration-Aware Prompt Learning for Medical Vision-Language Models
• E-BayesSAM: Efficient Bayesian Adaptation of SAM with Self-Optimizing KAN-Based Interpretation for Uncertainty-Aware Ultrasonic Segmentation
• Similarity-Distance-Magnitude Activations
• On the role of Model Uncertainties in Bayesian Optimization
• Protected Probabilistic Classification Library
• Quantum parameter estimation with uncertainty quantification from continuous measurement data using neural network ensembles
• Ensemble Distribution Distillation for Self-Supervised Human Activity Recognition
• Prior Distribution and Model Confidence
• Extracting Uncertainty Estimates from Mixtures of Experts for Semantic Segmentation
• What Fundamental Structure in Reward Functions Enables Efficient Sparse-Reward Learning?
• TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space
• Exploring accuracy and uncertainty quantification in physics-informed neural networks for inferring microbial community dynamics
• Diversity inducing Information Bottleneck in Model Ensembles
• Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal
• Domain Adaptation via Feature Refinement
• Entropy-Based Non-Invasive Reliability Monitoring of Convolutional Neural Networks
• Automated Quality Assessment for LLM-Based Complex Qualitative Coding: A Confidence-Diversity Framework
• Gradient Rectification for Robust Calibration under Distribution Shift
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• SNAP-UQ: Self-supervised Next-Activation Prediction for Single-Pass Uncertainty in TinyML
• TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML
• Resource-Aware Aggregation and Sparsification in Heterogeneous Ensemble Federated Learning
• Towards Robust Active Feature Acquisition
• Calibration Attention: Learning Reliability-Aware Representations for Vision Transformers
• Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking
• Unequal Uncertainty: Rethinking Algorithmic Interventions for Mitigating Discrimination from AI
• Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis
• Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring
• Infrared Object Detection with Ultra Small ConvNets: Is ImageNet Pretraining Still Useful?
• Model-Agnostic Dynamic Feature Selection with Uncertainty Quantification
• Bayesian Conformal Prediction via the Bayesian Bootstrap
• Semantic-Aware Gaussian Process Calibration with Structured Layerwise Kernels for Deep Neural Networks
• A Context-Aware Dual-Metric Framework for Confidence Estimation in Large Language Models
• Bayesian Neural Network Surrogates for Bayesian Optimization of Carbon Capture and Storage Operations
• Handling Out-of-Distribution Data: A Survey
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
• Explainability through uncertainty: Trustworthy decision-making with neural networks
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Role of calibration in uncertainty-based referral for deep learning
• To Trust or Not to Trust: On Calibration in ML-based Resource Allocation for Wireless Networks
• Confidence Calibration in Vision-Language-Action Models
• Bayesian Deep Learning for Convective Initiation Nowcasting Uncertainty Estimation
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification
• Implicit Generative Modeling for Efficient Exploration
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Distribution-Free Uncertainty-Aware Virtual Sensing via Conformalized Neural Operators
• Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift
• Monitoring Risks in Test-Time Adaptation
• Where are we with calibration under dataset shift in image classification?
• Semantic Certainty Assessment in Vector Retrieval Systems: A Novel Framework for Embedding Quality Evaluation
• Weighted Ensemble Models Are Strong Continual Learners
• Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift
• MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift
• Golden Grain: Building a Secure and Decentralized Model Marketplace for MLaaS
• Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanations
• CausalPFN: Amortized Causal Effect Estimation via In-Context Learning
• Sensitivity Analysis of Image Classification Models using Generalized Polynomial Chaos
• Quantifying Uncertainty in the Presence of Distribution Shifts
• h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective
• CP²: Leveraging Geometry for Conformal Prediction via Canonicalization
• Uncertainty Estimation by Human Perception versus Neural Models
• Towards Reliable WMH Segmentation under Domain Shift: An Application Study using Maximum Entropy Regularization to Improve Uncertainty Estimation
• Robust Molecular Property Prediction via Densifying Scarce Labeled Data
• Beyond Overconfidence: Foundation Models Redefine Calibration in Deep Neural Networks
• Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'
• Enhancing decision support in crop production: Analyzing conformal prediction for uncertainty quantification
• Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness
• TRUST: Test-time Resource Utilization for Superior Trustworthiness
• Reliably Detecting Model Failures in Deployment Without Labels
• Uncertainty-guided active learning for surrogate prediction of stream-finishing wear fields
• Generalized Negative Correlation Learning for Deep Ensembling
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
• Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models
• Hydra: Preserving Ensemble Diversity for Model Distillation
• Automated tick classification using deep learning and its associated challenges in citizen science
• Position: Epistemic uncertainty estimation methods are fundamentally incomplete
• Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
• DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes
• Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
• Continual Learning Beyond Experience Rehearsal and Full Model Surrogates
• Revisiting Bayesian Model Averaging in the Era of Foundation Models
• CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment
• Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection
• WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shifts
• Towards Machine Learning-based Model Predictive Control for HVAC Control in Multi-Context Buildings at Scale via Ensemble Learning
• Asymmetric Duos: Sidekicks Improve Uncertainty
• Decision Theoretic Bootstrapping
• EvidenceMoE: A Physics-Guided Mixture-of-Experts with Evidential Critics for Advancing Fluorescence Light Detection and Ranging in Scattering Media
• Constrained Co-Design for Photonic Bayesian Neural Networks
• Safe Uncertainty-Aware Learning of Robotic Suturing
• Robust Invariant Representation Learning by Distribution Extrapolation
• ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning
• Diving into the Fusion of Monocular Priors for Generalized Stereo Matching
• Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift
• HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing
• Imagination-Limited Q-Learning for Offline Reinforcement Learning
• EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning
• Are vision language models robust to uncertain inputs?
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors
• On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift
• Variational Visual Question Answering for Uncertainty-Aware Selective Prediction
• Contrastive Normalizing Flows for Uncertainty-Aware Parameter Estimation
• Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference
• Validation of Conformal Prediction in Cervical Atypia Classification
• Continuous Visual Autoregressive Generation via Score Maximization
• Performance Estimation in Binary Classification Using Calibrated Confidence
• Uncertainty Quantification for Machine Learning in Healthcare: A Survey
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
• Me, Myself, and π : Evaluating and Explaining LLM Introspection
• CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
• Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space
• Self-Aware Object Detection via Degradation Manifolds
• A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift
• Towards Automated Discovery: A Review of Generative Models, Multimodal Learning and Closed-Loop Workflows in Inverse Materials Design
• Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
• Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review
• Reliability-Aware ETF Tail-Risk Monitoring
• EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning
• Semi-Supervised Class Discovery
• An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression
• Evaluating Uncertainty in Deep Gaussian Processes
• When Is a Conformal Guarantee Fair? Auditing Silent Subgroup Under-Coverage in Alzheimer's Disease Longitudinal Prediction
• Explaining Low Perception Model Competency with High-Competency Counterfactuals
• Offline Robotic World Model: Learning Robotic Policies without a Physics Simulator
• Last-layer committee machines for uncertainty estimations of benthic imagery
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation
• Beyond One-Hot Labels: Semantic Mixing for Model Calibration
• Fully probabilistic quasar continua predictions near Lyman-α with conditional neural spline flows
• VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning
• SEAM: Global consistency beyond local accuracy in scientific machine learning
• Validity, Reliability, and Transparency in Artificial Intelligence Regulation
• A Unified Risk View of Uncertainty: Posterior Risk for Disentanglement and Evaluation Beyond Proxies
• SkillTFM: Gated Skill Evolution for Training-Free Adaptation of Tabular Foundation Models
• Evidential Rule Learning for Interpretable Classification with Abstention
• Balancing Two Classifiers via A Simplex ETF Structure for Model Calibration
• A review on deep learning for vision-based hand detection, hand segmentation and hand gesture recognition in human–robot interaction
• Quantifying uncertainty in machine learning on nuclear binding energy
• Road Grip Uncertainty Estimation Through Surface State Segmentation
• S2R-HDR: A Large-Scale Rendered Dataset for HDR Fusion
• Are We Done with Object-Centric Learning?
• Explaining Uncertainty in Multiple Sclerosis Lesion Segmentation Beyond Prediction Errors
• The challenge of uncertainty quantification of large language models in medicine
• Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Models

Related

• Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models
• A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learning
• Uncertainty quantification for trustworthy deep learning: Methods and measures
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Uncertainty Quantification for Deep Learning