Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?

2024/07/17 by Aman Sinha, Sinha, Aman, Timothee Mickus +6
Biochemistry, Genetics and Molecular Biology · #Biomedical Text Mining and Ontologies

paper · pdf · doi:10.48550/arxiv.2407.12626

Abstract

The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in-chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.

Related

• Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?
• The challenge of uncertainty quantification of large language models in medicine
• Label-Confidence-Aware Uncertainty Estimation in Natural Language Generation
• Distinguishing the Knowable from the Unknowable with Language Models
• Mitigating Uncertainty in Document Classification