Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not

2021/12/13 by Chelsea Murray, Murray, Chelsea, James Urquhart Allingham +5
Computer Science · Engineering · #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2112.06926

openalex publication_date 2021/12/13 · openalex created_date 2021/12/31 · openalex updated_date 2026/07/28

Abstract

Farquhar et al. [2021] show that correcting for active learning bias with underparameterised models leads to improved downstream performance. For overparameterised models such as NNs, however, correction leads either to decreased or unchanged performance. They suggest that this is due to an "overfitting bias" which offsets the active learning bias. We show that depth uncertainty networks operate in a low overfitting regime, much like underparameterised models. They should therefore see an increase in performance with bias correction. Surprisingly, they do not. We propose that this negative result, as well as the results Farquhar et al. [2021], can be explained via the lens of the bias-variance decomposition of generalisation error.

Related

• Depth Uncertainty Networks for Active Learning
• On Statistical Bias In Active Learning: How and When To Fix It
• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
• Random Bias Initialization Improves Quantized Training
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks