2022/05/16 by T. Takahashi, Takashi Takahashi, Takahashi, Takashi · 2 citations
Computer Science · Mathematics · Physics and Astronomy · #Algorithm #Artificial intelligence #Binary classification #Classifier (UML) #Computer science #Face and Expression Recognition #Gaussian #Heuristics #Linear classifier #Machine Learning and Data Classification #Machine learning #Mathematical optimization #Mathematics #Pattern recognition (psychology) #Support vector machine #cond-mat.dis-nn #cond-mat.stat-mech #cs.LG #math.ST #stat.ML #stat.TH
paper · pdf · doi:10.48550/arxiv.2205.07739
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2022/05/16 · openalex created_date 2022/05/22 · openalex updated_date 2026/08/05
Self-training (ST) is a simple yet effective semi-supervised learning method. However, why and how ST improves generalization performance by using potentially erroneous pseudo-labels is still not well understood. To deepen the understanding of ST, we derive and analyze a sharp characterization of the behavior of iterative ST when training a linear classifier by minimizing the ridge-regularized convex loss on binary Gaussian mixtures, in the asymptotic limit where input dimension and data size diverge proportionally. The results show that ST improves generalization in different ways depending on the number of iterations. When the number of iterations is small, ST improves generalization performance by fitting the model to relatively reliable pseudo-labels and updating the model parameters by a large amount at each iteration. This suggests that ST works intuitively. On the other hand, with many iterations, ST can gradually improve the direction of the classification plane by updating the model parameters incrementally, using soft labels and small regularization. It is argued that this is because the small update of ST can extract information from the data in an almost noiseless way. However, in the presence of label imbalance, the generalization performance of ST underperforms supervised learning with true labels. To overcome this, two heuristics are proposed to enable ST to achieve nearly compatible performance with supervised learning even with significant label imbalance.