2022/01/13 by K L Navaneet, Navaneet, K L, Soroush Abbasi Koohpayegani +5 · 5 citations
Computer Science · Engineering · Mathematics · #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Distillation #Domain Adaptation and Few-Shot Learning #Engineering #FOS: Computer and information sciences #Feature (linguistics) #Inference #Machine learning #Mathematics #Neural Networks and Applications #Pattern recognition (psychology) #Perceptron #Projection (relational algebra) #Regression #Simple (philosophy) #Statistics #Task (project management) #cs.CV
paper · pdf · doi:10.48550/arxiv.2201.05131
published in arXiv (Cornell University) (Cornell University) · In BMVC 2021. Code available at: https://github.com/UCDvision/simreg
arxiv created 2022/01/13 · openalex publication_date 2022/01/13 · arxiv updated 2022/01/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Feature regression is a simple way to distill large neural network models to smaller ones. We show that with simple changes to the network architecture, regression can outperform more complex state-of-the-art approaches for knowledge distillation from self-supervised models. Surprisingly, the addition of a multi-layer perceptron head to the CNN backbone is beneficial even if used only during distillation and discarded in the downstream task. Deeper non-linear projections can thus be used to accurately mimic the teacher without changing inference architecture and time. Moreover, we utilize independent projection heads to simultaneously distill multiple teacher networks. We also find that using the same weakly augmented image as input for both teacher and student networks aids distillation. Experiments on ImageNet dataset demonstrate the efficacy of the proposed changes in various self-supervised distillation settings.