2022/01/03 by Made Nindyatama Nityasya, Nityasya, Made Nindyatama, Haryo Akbarianto Wibowo +7 · 2 citations
Computer Science · #68T50 #Advanced Neural Network Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.6 #I.2.7 #Machine Learning and Data Classification #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2201.00558
openalex publication_date 2022/01/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04
We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our experiment involves 12 datasets grouped in two tasks: text classification and sequence labeling in the Indonesian language. We also compare various aspects of distillations including the usage of word embeddings and unlabeled data augmentation. Our experiments show that, despite the rising popularity of Transformer-based models, using BiLSTM and CNN student models provide the best trade-off between performance and computational resource (CPU, RAM, and storage) compared to pruned BERT models. We further propose some quick wins on performing KD to produce small NLP models via efficient KD training mechanisms involving simple choices of loss functions, word embeddings, and unlabeled data preparation.