2022/08/20 by Rajiv Movva, Jinhao Lei, Movva, Rajiv +7
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2208.09684
openalex publication_date 2022/08/20 · openalex created_date 2022/08/24 · openalex updated_date 2026/07/28
Quantization, knowledge distillation, and magnitude pruning are among the most popular methods for neural network compression in NLP. Independently, these methods reduce model size and can accelerate inference, but their relative benefit and combinatorial interactions have not been rigorously studied. For each of the eight possible subsets of these techniques, we compare accuracy vs. model size tradeoffs across six BERT architecture sizes and eight GLUE tasks. We find that quantization and distillation consistently provide greater benefit than pruning. Surprisingly, except for the pair of pruning and quantization, using multiple methods together rarely yields diminishing returns. Instead, we observe complementary and super-multiplicative reductions to model size. Our work quantitatively demonstrates that combining compression methods can synergistically reduce model size, and that practitioners should prioritize (1) quantization, (2) knowledge distillation, and (3) pruning to maximize accuracy vs. model size tradeoffs.