Smooth Neighbors on Teacher Graphs for Semi-supervised Learning

2017/11/01 by Yucen Luo, Luo, Yucen, Jun Zhu +7 · 1 citation
Computer Science · Mathematics · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1711.00258

Accept as Spotlight in Computer Vision and Pattern Recognition 2018

openalex publication_date 2017/11/01 · arxiv created 2018/03/28 · arxiv updated 2018/03/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data samples. In this paper, we propose a novel method, called Smooth Neighbors on Teacher Graphs (SNTG). In SNTG, a graph is constructed based on the predictions of the teacher model, i.e., the implicit self-ensemble of models. Then the graph serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold. We achieve state-of-the-art results on semi-supervised learning benchmarks. The error rates are 9.89%, 3.99% for CIFAR-10 with 4000 labels, SVHN with 500 labels, respectively. In particular, the improvements are significant when the labels are fewer. For the non-augmented MNIST with only 20 labels, the error rate is reduced from previous 4.81% to 1.36%. Our method also shows robustness to noisy labels.

Citations

• Adam: A Method for Stochastic Optimization
• Improving neural networks by preventing co-adaptation of feature detectors
• Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Improved Techniques for Training GANs
• Triple Generative Adversarial Nets
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Good Semi-supervised Learning that Requires a Bad GAN
• Temporal Ensembling for Semi-Supervised Learning
• CNN Features off-the-shelf: an Astounding Baseline for Recognition

Cited by

• TRiCo: Triadic Game-Theoretic Co-Training for Robust Semi-Supervised Learning

Related

• Local Clustering with Mean Teacher for Semi-supervised Learning
• Every Node Counts: Self-Ensembling Graph Convolutional Networks for Semi-Supervised Learning
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Regularizing Semi-supervised Graph Convolutional Networks with a Manifold Smoothness Loss
• CompRess: Self-Supervised Learning by Compressing Representations