2020/10/14 by Artem Chumachenko, Chumachenko, Artem, Daniil Gavrilov +5
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2010.06993
openalex publication_date 2020/10/14 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
In this work, we present a novel approach for simultaneous knowledge transfer\nand model compression called Weight Squeezing. With this method, we perform\nknowledge transfer from a teacher model by learning the mapping from its\nweights to smaller student model weights.\n We applied Weight Squeezing to a pre-trained text classification model based\non BERT-Medium model and compared our method to various other knowledge\ntransfer and model compression methods on GLUE multitask benchmark. We observed\nthat our approach produces better results while being significantly faster than\nother methods for training student models.\n We also proposed a variant of Weight Squeezing called Gated Weight Squeezing,\nfor which we combined fine-tuning of BERT-Medium model and learning mapping\nfrom BERT-Base weights. We showed that fine-tuning with Gated Weight Squeezing\noutperforms plain fine-tuning of BERT-Medium model as well as other concurrent\nSoTA approaches while much being easier to implement.\n