Self-calibration for Language Model Quantization and Pruning

2024/10/22 by Miles Williams, Williams, Miles, George Chrysostomou +4 · 1 voice · 4 citations
Computer Science · #Natural Language Processing Techniques #cs.CL

paper · pdf · doi:10.48550/arxiv.2410.17170

openalex publication_date 2024/10/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Quantization and pruning are fundamental approaches for model compression, enabling efficient inference for language models. In a post-training setting, state-of-the-art quantization and pruning methods require calibration data, a small set of unlabeled examples. Conventionally, this is randomly sampled web text, aiming to reflect the model training data. However, this poses two key problems: (1) unrepresentative calibration examples can harm model performance, and (2) organizations increasingly avoid releasing model training data. In this paper, we propose self-calibration as a solution. Our approach requires no external data, instead leveraging the model itself to generate synthetic calibration data, with a view to better approximating the pre-training data distribution. We extensively compare the performance of self-calibration with several baselines, across a variety of models, compression methods, and tasks. Our approach proves consistently competitive in maximizing downstream task performance, frequently outperforming even using real data.

Cited by

• Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models
• Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning
• Fair-GPTQ: Bias-Aware Quantization for Large Language Models
• Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction

Discussions

• Synthetic calibration data (for pruning and quantization) generated by the LLM itself is a better approx of the pre-training data dist than "external" data. Really cool work by Miles (@mileswil.bsky [bsky, 5 points, 0 comments]

Related

• Beware of Calibration Data for Pruning Large Language Models
• You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
• Making Pre-trained Language Models both Task-solvers and Self-calibrators
• Rethinking Post-Training Quantization: Introducing a Statistical Pre-Calibration Approach
• Pruning Pre-trained Language Models with Principled Importance and Self-regularization