Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?

2022/09/07 by Wimmer, Lisa, Sale, Yusuf, Hofman, Paul +2 · 23 citations
Computer Science · Materials Science · #Adversarial Robustness in Machine Learning #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science

paper · pdf · doi:10.48550/arxiv.2209.03302

openalex publication_date 2022/09/07 · openalex created_date 2022/09/30 · openalex updated_date 2026/07/28

Abstract

The quantification of aleatoric and epistemic uncertainty in terms of conditional entropy and mutual information, respectively, has recently become quite common in machine learning. While the properties of these measures, which are rooted in information theory, seem appealing at first glance, we identify various incoherencies that call their appropriateness into question. In addition to the measures themselves, we critically discuss the idea of an additive decomposition of total uncertainty into its aleatoric and epistemic constituents. Experiments across different computer vision tasks support our theoretical findings and raise concerns about current practice in uncertainty quantification.

Cited by

• Epistemic Reject Option Prediction
• An Isotropic Approach to Efficient Uncertainty Quantification with Gradient Norms
• Evaluating large language models for diagnostic reasoning from unstructured clinical narratives in epilepsy
• Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification
• Uncertainty Quantification for Machine Learning: One Size Does Not Fit All
• Beyond Accuracy: An Empirical Study of Uncertainty Estimation in Imputation
• The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguity
• Uncertainty Quantification for Regression: A Unified Framework based on kernel scores
• Structurally Separated Uncertainty in Supervised Latent Variable Models
• Frequentist Validity of Epistemic Uncertainty Estimators
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning
• On Uncertainty Calibration for Equivariant Functions
• Trading with the Devil: Risk and Return in Foundation Model Strategies
• HybridFlow: Quantification of Aleatoric and Epistemic Uncertainty with a Single Hybrid Model
• Uncertainty Quantification for Regression using Proper Scoring Rules
• Tracing Uncertainty in Language Model "Reasoning"
• The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models
• Multidimensional Uncertainty Quantification via Optimal Transport
• Uncertainty-Aware Knowledge Tracing Models
• Uncertainty Estimation using Variance-Gated Distributions
• Uncertainty Quantification and Propagation for ACORN, a geometric deep learning tracking pipeline for HEP experiments
• Model-Agnostic Dynamic Feature Selection with Uncertainty Quantification
• Decision Theory For Large Scale Outlier Detection Using Aleatoric Uncertainty: With a Note on Bayesian FDR

Related

• Rethinking Aleatoric and Epistemic Uncertainty
• Uncertainty Quantification for Machine Learning: One Size Does Not Fit All
• Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' View
• Epistemic Uncertainty Sampling
• Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Tasks