Being Bayesian about Categorical Probability

2020/02/19 by Taejong Joo, Joo, Taejong, Uijung Chung +3 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2002.07965

openalex publication_date 2020/02/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural networks utilize the softmax as a building block in classification tasks, which contains an overconfidence problem and lacks an uncertainty representation ability. As a Bayesian alternative to the softmax, we consider a random variable of a categorical probability over class labels. In this framework, the prior distribution explicitly models the presumed noise inherent in the observed label, which provides consistent gains in generalization performance in multiple challenging tasks. The proposed method inherits advantages of Bayesian approaches that achieve better uncertainty estimation and model calibration. Our method can be implemented as a plug-and-play loss function with negligible computational overhead compared to the softmax with the cross-entropy loss function.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Natural Gradient Works Efficiently in Learning
• A Practical Bayesian Framework for Backpropagation Networks
• ImageNet Large Scale Visual Recognition Challenge
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Weight Uncertainty in Neural Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• MedBayes-Lite: A Clinical Uncertainty Governance Layer for Risk-Aware Medical Decision Support
• Disturbing Target Values for Neural Network Regularization
• Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer

Related

• Fast Predictive Uncertainty for Classification with Bayesian Deep\n Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models
• Bayesian neural networks with interpretable priors from Mercer kernels