2021/04/19 by Pengguang Chen, Shu Liu, Chen, Pengguang +5 · 53 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial intelligence #Code (set theory) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Construct (python library) #Distillation #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Feature (linguistics) #Focus (optics) #Machine learning #Overhead (engineering) #Path (computing) #Programming language #Transformation (genetics) #Variety (cybernetics) #cs.CV
paper · pdf · doi:10.48550/arxiv.2104.09044
published in arXiv (Cornell University) (Cornell University) · CVPR 2021
arxiv created 2021/04/19 · openalex publication_date 2021/04/19 · arxiv updated 2021/04/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08
Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network. Previous methods mostly focus on proposing feature transformation and loss functions between the same level's features to improve the effectiveness. We differently study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our new review mechanism is effective and structurally simple. Our finally designed nested and compact framework requires negligible computation overhead, and outperforms other methods on a variety of tasks. We apply our method to classification, object detection, and instance segmentation tasks. All of them witness significant student network performance improvement. Code is available at https://github.com/Jia-Research-Lab/ReviewKD