2025/01/31 by Sixiao Huang, Tintin Wang, Huang, Sixiao +12
Computer Science · Engineering · Mathematics · #Algorithms and Data Compression #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Power Systems and Technologies #Tensor decomposition and applications
paper · pdf · doi:10.48550/arxiv.2501.19135
openalex publication_date 2025/01/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Large language models (LLMs) are both storage-intensive and computation-intensive, posing significant challenges when deployed on resource-constrained hardware. As linear layers in LLMs are mainly resource consuming parts, this paper develops a tensor-train decomposition (TTD) for LLMs with a further hardware implementation on FPGA. TTD compression is applied to the linear layers in ChatGLM3-6B and LLaMA2-7B models with compression ratios (CRs) for the whole network 1.94× and 1.60×, respectively. The compressed LLMs are further implemented on FPGA hardware within a highly efficient group vector systolic array (GVSA) architecture, which has DSP-shared parallel vector PEs for TTD inference, as well as optimized data communication in the accelerator. Experimental results show that the corresponding TTD based LLM accelerator implemented on FPGA achieves 1.45× and 1.57× reduction in first token delay for ChatGLM3-6B and LLaMA2-7B models, respectively.