2024/01/29 by V. Abronin, Abronin, V., A. Naumov +17 · 1 citation
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computational Physics and Python Applications #Computer Science and Engineering #FOS: Computer and information sciences #Machine Learning (cs.LG) #Tensor decomposition and applications
paper · pdf · doi:10.48550/arxiv.2401.16367
openalex publication_date 2024/01/29 · openalex created_date 2024/01/31 · openalex updated_date 2026/07/28
We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associated with factorization. We demonstrate this method applied to the GPT-2small. The result of the compression is TQCompressedGPT-2 model, featuring 81 mln. parameters compared to 124 mln. in the GPT-2small. We make TQCompressedGPT-2 publicly available. We further enhance the performance of the TQCompressedGPT-2 through a training strategy involving multi-step knowledge distillation, using only a 3.1% of the OpenWebText. TQCompressedGPT-2 surpasses DistilGPT-2 and KnGPT-2 in comparative evaluations, marking an advancement in the efficient and effective deployment of models in resource-constrained environments.