2023/01/27 by Aref Jafari, Mehdi Rezagholizadeh, Jafari, Aref +3
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural Networks and Applications #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2301.12006
openalex publication_date 2023/01/27 · openalex created_date 2023/02/01 · openalex updated_date 2026/07/28
Knowledge distillation (KD) is one of the prominent techniques for model\ncompression. In this method, the knowledge of a large network (teacher) is\ndistilled into a model (student) with usually significantly fewer parameters.\nKD tries to better-match the output of the student model to that of the teacher\nmodel based on the knowledge extracts from the forward pass of the teacher\nnetwork. Although conventional KD is effective for matching the two networks\nover the given data points, there is no guarantee that these models would match\nin other areas for which we do not have enough training samples. In this work,\nwe address that problem by generating new auxiliary training samples based on\nextracting knowledge from the backward pass of the teacher in the areas where\nthe student diverges greatly from the teacher. We compute the difference\nbetween the teacher and the student and generate new data samples that maximize\nthe divergence. This is done by perturbing data samples in the direction of the\ngradient of the difference between the student and the teacher. Augmenting the\ntraining set by adding this auxiliary improves the performance of KD\nsignificantly and leads to a closer match between the student and the teacher.\nUsing this approach, when data samples come from a discrete domain, such as\napplications of natural language processing (NLP) and language understanding,\nis not trivial. However, we show how this technique can be used successfully in\nsuch applications. We evaluated the performance of our method on various tasks\nin computer vision and NLP domains and got promising results.\n