NIRVANA: Structured Pruning Reimagined for Large Language Model Compression

2025/09/17 by Mengting Ai, Ai, Mengting, Tianxin Wei +5 · 1 citation
Computer Science · #Calibration #Code (set theory) #Domain Adaptation and Few-Shot Learning #Heuristics #Kernel (algebra) #Language model #Multimodal Machine Learning Applications #Pruning #Sensitivity (control systems) #Structured prediction #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.2509.14230

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/09/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities. To overcome these barriers, we present NIRVANA, a novel, hardware-aware structured pruning framework designed to preserve both zero-shot performance and the optimization landscape for downstream fine-tuning. Departing from traditional loss-based heuristics, our approach evaluates structural importance through a first-order function-space saliency inspired by the Neural Tangent Kernel (NTK), effectively safeguarding the model's critical training dynamics. To prevent structural collapse at high compression rates, we introduce a global unit-ranking strategy coupled with an analytically derived allocation mechanism, which optimally balances the pruning aggressiveness between attention heads and MLP neurons. Furthermore, we eliminate the instability typically associated with random data sampling by employing a lightweight, KL-divergence-driven calibration data selection process. Extensive evaluations across Llama3, Qwen, and T5 architectures demonstrate that NIRVANA consistently establishes new state-of-the-art results on different benchmarks, providing a theoretically sound and practical approach to LLM compression. The code is available at https://github.com/iDEA-iSAIL-Lab-UIUC/NIRVANA.

Citations

• A Simple and Effective Pruning Approach for Large Language Models
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• Program Synthesis with Large Language Models
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• LoRA: Low-Rank Adaptation of Large Language Models
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Attention Is All You Need

Cited by

• Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

Related

• LLM-Pruner: On the Structural Pruning of Large Language Models
• Fluctuation-based Adaptive Structured Pruning for Large Language Models
• You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
• Aggressive Post-Training Compression on Extremely Large Language Models
• Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models