Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning

2026/06/30 by Hu Xu, Zhaolong Xing, Congcong Liu +5
Computer Science · #cs.LG #cs.AI

paper · pdf

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Calibration data are often treated as a minor implementation detail in post-training LLM pruning because averaged evaluations suggest only modest effects. We show that this conclusion is an averaging artifact: at 60% SparseGPT sparsity, calibration strategies separated by only 2.85 points in averaged commonsense accuracy differ by 51.9 points in Code retention. Across 15 sources, capability-decomposed analysis reveals an opposing pattern: calibration perplexity is positively associated with General retention but negatively associated with Math or Code retention, leaving no evaluated single source uniformly strong across capabilities. This finding motivates capability-balanced multi-source calibration. Under the same calibration budget, a balanced real-data mixture outperforms every evaluated single source on LLaMA-3.1-8B, beating C4 by 18.8 points; the advantage grows with sparsity and persists on LLaMA-3.1-70B. Because the original pretraining data of advanced LLMs are often inaccessible, we further introduce Information-Guided Self-Calibration for Pruning (IGSP). Using only the base model and evaluation taxonomy, IGSP generates capability-stratified pools and selects low-redundancy samples within capability-specific perplexity ranges, outperforming Self-Cal and SGS by up to 4.8 points. Together, these results recast calibration as a capability-coverage problem and identify multi-source design as a practical principle for preserving capabilities in high-sparsity LLM pruning.

Citations

• General Information Metrics for Improving AI Model Training Efficiency
• Self-calibration for Language Model Quantization and Pruning
• Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning
• On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning
• DataComp-LM: In search of the next generation of training sets for language models
• Compressing LLMs: The Truth is Rarely Pure and Never Simple
• Do Compressed LLMs Forget Knowledge? An Experimental Study with Practical Implications
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Solving Quantitative Reasoning Problems with Language Models
• Training Verifiers to Solve Math Word Problems
• PIQA: Reasoning about Physical Commonsense in Natural Language
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• The LAMBADA dataset: Word prediction requiring a broad discourse context

Related

• Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning
• Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
• Beware of Calibration Data for Pruning Large Language Models
• Multicalibration for LLM-based Code Generation
• Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs