Mitigating Uncertainty in Document Classification

2019/07/17 by Xuchao Zhang, Fanglan Chen, Zhang, Xuchao +6 · 3 citations
Computer Science · Mathematics · #Anomaly Detection Techniques and Applications #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1907.07590

Accepted by NAACL19

arxiv created 2019/07/17 · openalex publication_date 2019/07/17 · arxiv updated 2019/07/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The uncertainty measurement of classifiers' predictions is especially important in applications such as medical diagnoses that need to ensure limited human resources can focus on the most uncertain predictions returned by machine learning models. However, few existing uncertainty models attempt to improve overall prediction accuracy where human resources are involved in the text classification task. In this paper, we propose a novel neural-network-based model that applies a new dropout-entropy method for uncertainty measurement. We also design a metric learning method on feature representations, which can boost the performance of dropout-based uncertainty methods with smaller prediction variance in accurate prediction trials. Extensive experiments on real-world data sets demonstrate that our method can achieve a considerable improvement in overall prediction accuracy compared to existing approaches. In particular, our model improved the accuracy from 0.78 to 0.92 when 30% of the most uncertain predictions were handed over to human experts in "20NewsGroup" data.

Citations

• Adam: A Method for Stochastic Optimization
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Distance-based Confidence Score for Neural Network Classifiers
• Deep Neural Networks as Gaussian Processes
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• A Survey on Bayesian Deep Learning

Cited by

• Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Heads
• An Interpretable and Uncertainty Aware Multi-Task Framework for Multi-Aspect Sentiment Analysis
• A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs

Related

• Revisiting Softmax for Uncertainty Approximation in Text Classification
• Nonparametric Uncertainty Quantification for Single Deterministic Neural Network
• Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?
• Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers
• Confidence estimation of classification based on the distribution of the neural network output layer