Knowledge distillation for optimization of quantized deep neural networks

2019/09/04 by Sungho Shin, Shin, Sungho, Yoonho Boo +3
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1909.01688

openalex publication_date 2019/09/04 · openalex created_date 2019/11/01 · openalex updated_date 2026/07/28

Abstract

Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coefficient, and the size of teacher network for QDNN training. We analyze the effect of these hyper-parameters for QDNN optimization with KD. We find that these hyper-parameters are inter-related, and also introduce a simple and effective technique that reduces coefficient during training. With KD employing the proposed hyper-parameters, we achieve the test accuracy of 92.7% and 67.0% on Resnet20 with 2-bit ternary weights for CIFAR-10 and CIFAR-100 data sets, respectively.

Citations

• Distilling the Knowledge in a Neural Network
• Deep Learning with Limited Numerical Precision
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Resiliency of Deep Neural Networks under Quantization
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Recurrent Neural Networks With Limited Numerical Precision
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Low Precision RNNs: Quantizing RNNs Without Losing Accuracy
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

Related

• QKD: Quantization-aware Knowledge Distillation
• Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data
• Data-Augmented Quantization-Aware Knowledge Distillation
• Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation
• Weight Distillation: Transferring the Knowledge in Neural Network Parameters