Inhibited Softmax for Uncertainty Estimation in Neural Networks

2018/10/03 by Marcin Możejko, Możejko, Marcin, Mateusz Susik +3 · 24 citations
Computer Science · Engineering · Mathematics · Physics and Astronomy · #Adversarial Robustness in Machine Learning #Algorithm #Anomaly Detection Techniques and Applications #Artificial intelligence #Artificial neural network #Computer science #Constant (computer programming) #Data mining #Engineering #Estimation #FOS: Computer and information sciences #Image (mathematics) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Pattern recognition (psychology) #Preprocessor #Softmax function #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1810.01861

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2018/10/03 · arxiv created 2019/04/07 · arxiv updated 2019/04/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a new method for uncertainty estimation and out-of-distribution detection in neural networks with softmax output. We extend softmax layer with an additional constant input. The corresponding additional output is able to represent the uncertainty of the network. The proposed method requires neither additional parameters nor multiple forward passes nor input preprocessing nor out-of-distribution datasets. We show that our method performs comparably to more computationally expensive methods and outperforms baselines on our experiments from image recognition and sentiment analysis domains.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Bayesian Dark Knowledge
• Gradient-based learning applied to document recognition
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Quantifying Uncertainty in Machine Learning-Based Pervasive Systems: Application to Human Activity Recognition
• Long-Term Alzheimers Disease Prediction: A Novel Image Generation Method Using Temporal Parameter Estimation with Normal Inverse Gamma Distribution on Uneven Time Series
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving

Related

• Understanding Softmax Confidence and Uncertainty
• Fine-grained Uncertainty Modeling in Neural Networks
• Explorations of the Softmax Space: Knowing When the Neural Network Doesn't Know
• Softmax Is Not an Artificial Trick: An Information-Theoretic View of Softmax in Neural Networks
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks