Efficient GPT Model Pre-training using Tensor Train Matrix Representation

2023/06/05 by Viktoriia Chekalina, Georgii Novikov, Chekalina, Viktoriia +7 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computational Physics and Python Applications #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2306.02697

openalex publication_date 2023/06/05 · openalex created_date 2023/06/07 · openalex updated_date 2026/07/28

Abstract

Large-scale transformer models have shown remarkable performance in language modelling tasks. However, such models feature billions of parameters, leading to difficulties in their deployment and prohibitive training costs from scratch. To reduce the number of the parameters in the GPT-2 architecture, we replace the matrices of fully-connected layers with the corresponding Tensor Train Matrix~(TTM) structure. Finally, we customize forward and backward operations through the TTM-based layer for simplicity and the stableness of further training. % The resulting GPT-2-based model stores up to 40% fewer parameters, showing the perplexity comparable to the original model. On the downstream tasks, including language understanding and text summarization, the model performs similarly to the original GPT-2 model. The proposed tensorized layers could be used to efficiently pre-training other Transformer models.

Cited by

• TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices

Related

• Primer: Searching for Efficient Transformers for Language Modeling
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
• TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
• LightSeq2: Accelerated Training for Transformer-based Models on GPUs
• Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models