Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework

2021/10/04 by Tengteng Huang, Yifan Sun, Huang, Tengteng +7 · 5 citations
Computer Science · #Advanced Neural Network Applications #Artificial intelligence #Class (philosophy) #Complementarity (molecular biology) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Domain Adaptation and Few-Shot Learning #Ensemble learning #FOS: Computer and information sciences #Machine Learning and ELM #Machine learning #Smoothing #cs.CV

paper · pdf · doi:10.48550/arxiv.2110.01253

published in arXiv (Cornell University) (Cornell University)

arxiv created 2021/10/04 · openalex publication_date 2021/10/04 · arxiv updated 2021/10/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model smoothing is of central importance for obtaining a reliable teacher model in the student-teacher framework, where the teacher generates surrogate supervision signals to train the student. A popular model smoothing method is the Temporal Moving Average (TMA), which continuously averages the teacher parameters with the up-to-date student parameters. In this paper, we propose "Spatial Ensemble", a novel model smoothing mechanism in parallel with TMA. Spatial Ensemble randomly picks up a small fragment of the student model to directly replace the corresponding fragment of the teacher model. Consequentially, it stitches different fragments of historical student models into a unity, yielding the "Spatial Ensemble" effect. Spatial Ensemble obtains comparable student-teacher learning performance by itself and demonstrates valuable complementarity with temporal moving average. Their integration, named Spatial-Temporal Smoothing, brings general (sometimes significant) improvement to the student-teacher learning framework on a variety of state-of-the-art methods. For example, based on the self-supervised method BYOL, it yields +0.9% top-1 accuracy improvement on ImageNet, while based on the semi-supervised approach FixMatch, it increases the top-1 accuracy by around +6% on CIFAR-10 when only few training labels are available. Codes and models are available at: https://github.com/tengteng95/Spatial_Ensemble.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• A Simple Framework for Contrastive Learning of Visual Representations
• Prototypical Contrastive Learning of Unsupervised Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Big Self-Supervised Models are Strong Semi-Supervised Learners

Related

• See Further When Clear: Curriculum Consistency Model
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
• Student-Teacher Learning from Clean Inputs to Noisy Inputs
• Switching Temporary Teachers for Semi-Supervised Semantic Segmentation