Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning

2019/01/17 by Lu Liu, Liu, Lu, Robby T. Tan +1 · 1 citation
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1901.05657

openalex publication_date 2019/01/17 · openalex created_date 2021/05/24 · openalex updated_date 2026/07/28

Abstract

One of the successful approaches in semi-supervised learning is based on the consistency regularization. Typically, a student model is trained to be consistent with teacher prediction for the inputs under different perturbations. To be successful, the prediction targets given by teacher should have good quality, otherwise the student can be misled by teacher. Unfortunately, existing methods do not assess the quality of the teacher targets. In this paper, we propose a novel Certainty-driven Consistency Loss (CCL) that exploits the predictive uncertainty in the consistency loss to let the student dynamically learn from reliable targets. Specifically, we propose two approaches, i.e. Filtering CCL and Temperature CCL to either filter out uncertain predictions or pay less attention on them in the consistency regularization. We further introduce a novel decoupled framework to encourage model difference. Experimental results on SVHN, CIFAR-10, and CIFAR-100 demonstrate the advantages of our method over a few existing methods.

Citations

• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Semi-Supervised Learning with Ladder Networks
• Distilling the Knowledge in a Neural Network
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Cited by

• Self-Reinforcing Prototype Evolution with Dual-Knowledge Cooperation for Semi-Supervised Lifelong Person Re-Identification

Related

• Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning
• Alleviating Robust Overfitting of Adversarial Training With Consistency Regularization
• Semi-Supervised Learning with Variational Bayesian Inference and Maximum Uncertainty Regularization
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Adaptive Consistency Regularization for Semi-Supervised Transfer Learning