DistiLLM: Towards Streamlined Distillation for Large Language Models

2024/02/06 by Jongwoo Ko, Ko, Jongwoo, Sungnyun Kim +5 · 18 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2402.03898

openalex publication_date 2024/02/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Knowledge distillation (KD) is widely used for compressing a teacher model to a smaller student model, reducing its inference cost and memory footprint while preserving model capabilities. However, current KD methods for auto-regressive sequence models (e.g., large language models) suffer from missing a standardized objective function. Moreover, the recent use of student-generated outputs to address training-inference mismatches has significantly escalated computational costs. To tackle these issues, we introduce DistiLLM, a more effective and efficient KD framework for auto-regressive language models. DistiLLM comprises two components: (1) a novel skew Kullback-Leibler divergence loss, where we unveil and leverage its theoretical properties, and (2) an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs. Extensive experiments, including instruction-following tasks, demonstrate the effectiveness of DistiLLM in building high-performing student models while achieving up to 4.3× speedup compared to recent KD methods.

Cited by

• Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
• An Online Fragmentation-Aware Scheduler for Managing GPU-Sharing Workloads on Multi-Instance GPUs
• MemLoRA: Distilling Expert Adapters for On-Device Memory Systems
• SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
• Unified Reinforcement and Imitation Learning for Vision-Language Models
• DistilLock: Safeguarding LLMs from Unauthorized Knowledge Distillation on the Edge
• Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
• BitNet Distillation
• LLM-Oriented Token-Adaptive Knowledge Distillation
• AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution
• Distillation of Large Language Models via Concrete Score Matching
• ORPO-Distill: Mixed-Policy Preference Optimization for Cross-Architecture LLM Distillation
• Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
• SPADE: Structured Pruning and Adaptive Distillation for Efficient LLM-TTS
• Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
• Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillation
• GPO: Learning from Critical Steps to Improve LLM Reasoning
• Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning

Related

• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models
• Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework
• Revisiting Knowledge Distillation for Autoregressive Language Models
• SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models