LLM Pruning and Distillation in Practice: The Minitron Approach

2024/08/21 by Sharath Turuvekere Sreenivas, Saurav Muralidharan, Sreenivas, Sharath Turuvekere +41 · 3 voices · 26 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Extraction and Separation Processes #FOS: Computer and information sciences #Machine Learning (cs.LG) #Metallurgical Processes and Thermodynamics #Mineral Processing and Grinding #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2408.11796

openalex publication_date 2024/08/21 · arxiv published 2024/08/21 · arxiv updated 2024/12/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation. We explore two distinct pruning strategies: (1) depth pruning and (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on common benchmarks from the LM Evaluation Harness. The models are then aligned with NeMo Aligner and tested in instruct-tuned versions. This approach produces a compelling 4B model from Llama 3.1 8B and a state-of-the-art Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo 12B. We found that with no access to the original data, it is beneficial to slightly fine-tune teacher models on the distillation dataset. We open-source our base model weights on Hugging Face with a permissive license.

Cited by

• Compressing LLMs with MoP: Mixture of Pruners
• Can abstract concepts from LLM improve SLM performance?
• CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
• Atom: Efficient On-Device Video-Language Pipelines Through Modular Reuse
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• Neodragon: Mobile Video Generation using Diffusion Transformer
• Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
• Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
• Ministral 3
• Iterative Layer Pruning for Efficient Translation Inference
• Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search
• AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
• Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
• Boomerang Distillation Enables Zero-Shot Model Size Interpolation
• Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial
• OverFill: Two-Stage Models for Efficient Language Model Decoding
• Systematic Evaluation of Optimization Techniques for Long-Context Language Models
• Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation
• FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
• Distilling Empathy from Large Language Models
• Efficient Interleaved Speech Modeling through Knowledge Distillation
• MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs
• Inference-Time Hyper-Scaling with KV Cache Compression
• The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective
• FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition

Discussions

• LLM Pruning and Distillation in Practice: The Minitron Approach [hn, 2 points, 1 comments]
• 🚀 NVIDIA Minitron: Efficient LLM Compression! arxiv.org/pdf/2408.11796 Minitron uses pruning + distillation to create smaller, high-performance models 🔑 Highlights: - Teacher Correction: Adapts [bsky, 1 points, 0 comments]
• LLM Pruning and Distillation in Practice: The Minitron Approach [hn, 1 points, 0 comments]

Related

• Compact Language Models via Pruning and Knowledge Distillation
• SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models
• Reassessing Layer Pruning in LLMs: New Insights and Methods
• Lillama: Large Language Models Compression via Low-Rank Feature Distillation
• MultiPruner: Balanced Structure Removal in Foundation Models