Aggressive Post-Training Compression on Extremely Large Language Models

2024/09/30 by Zhang, Zining, Chen, Yao, He, Bingsheng +1
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2409.20094

Abstract

The increasing size and complexity of Large Language Models (LLMs) pose challenges for their deployment on personal computers and mobile devices. Aggressive post-training model compression is necessary to reduce the models' size, but it often results in significant accuracy loss. To address this challenge, we propose a novel network pruning technology that utilizes over 0.7 sparsity and less than 8 bits of quantization. Our approach enables the compression of prevailing LLMs within a couple of hours while maintaining a relatively small accuracy loss. In experimental evaluations, our method demonstrates effectiveness and potential for practical deployment. By making LLMs available on domestic devices, our work can facilitate a new era of natural language processing applications with wide-ranging impacts.

Related

• Compact Language Models via Pruning and Knowledge Distillation
• One-for-All Pruning: A Universal Model for Customized Compression of Large Language Models
• A Simple and Effective Pruning Approach for Large Language Models
• LLM-Pruner: On the Structural Pruning of Large Language Models
• Streamlining Redundant Layers to Compress Large Language Models