Pruning General Large Language Models into Customized Expert Models

2025/06/03 by Guizhen Chen, Zhao, Yirao, Chen, Guizhen +6
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.02561

openalex publication_date 2025/06/03 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have revolutionized natural language processing, yet their substantial model sizes often require substantial computational resources. To preserve computing resources and accelerate inference speed, it is crucial to prune redundant parameters, especially for experienced users who often need compact expert models tailored to specific downstream scenarios. However, most existing pruning methods focus on preserving the model's general capabilities, often requiring extensive post-training or suffering from degraded performance due to coarse-grained pruning. In this work, we design a \underlineCustom \underlinePruning method (Cus-Prun) to prune a large general model into a smaller lightweight expert model, which is positioned along the "language", "domain" and "task" dimensions. By identifying and pruning irrelevant neurons of each dimension, Cus-Prun creates expert models without any post-training. Our experiments demonstrate that Cus-Prun consistently outperforms other methods, achieving minimal loss in both expert and general capabilities across various models from different model families and sizes.

Citations

• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Pruning Large Language Models with Semi-Structural Adaptive Sparse Training
• SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages
• Reconstruct the Pruned Model without Any Retraining
• Multilingual Knowledge Editing with Language-Agnostic Factual Neurons
• Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
• How do Large Language Models Handle Multilingualism?
• Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
• Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
• LaCo: Large Language Model Pruning via Layer Collapse
• SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns
• The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
• Llemma: An Open Language Model For Mathematics
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
• Large Language Models in Finance: A Survey
• Code Llama: Open Foundation Models for Code
• A Cost Analysis of Generative Language Models and Influence Operations
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Sentiment Analysis in the Era of Large Language Models: A Reality Check
• QLoRA: Efficient Finetuning of Quantized LLMs
• Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca
• GPT-4 Technical Report
• Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations
• Language Models are Multilingual Chain-of-Thought Reasoners
• A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction
• Parameter-Efficient Sparsity for Large Language Models Fine-Tuning
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
• Training Verifiers to Solve Math Word Problems
• FinQA: A Dataset of Numerical Reasoning over Financial Data
• XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
• I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews
• Open Question Answering over Tables and Text
• The Multilingual Amazon Reviews Corpus
• Measuring Massive Multitask Language Understanding
• On the Cross-lingual Transferability of Monolingual Representations
• XNLI: Evaluating Cross-lingual Sentence Representations
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional\n Neural Networks for Extreme Summarization
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Character-level Convolutional Networks for Text Classification
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Cited by

• Persona-Pruner: Sculpting Lightweight Models for Role-Playing

Related

• Large Language Model Pruning
• Automatic Pruning Discovery for Large Language Models
• SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models
• Frustratingly Easy Task-aware Pruning for Large Language Models
• Improved Methods for Model Pruning and Knowledge Distillation