Black-Box On-Policy Distillation of Large Language Models

2025/11/13 by Tianxiao Ye, Ye, Tianzhu, Dong Li +9 · 3 citations
Computer Science · #Topic Modeling #Explainable Artificial Intelligence (XAI) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2511.10643

Abstract

Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM's, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.

Citations

• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• OpenThoughts: Data Recipes for Reasoning Models
• Qwen3 Technical Report
• LIMO: Less is More for Reasoning
• s1: Simple test-time scaling
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Qwen2.5 Technical Report
• The Llama 3 Herd of Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Instruction Tuning with GPT-4
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Training language models to follow instructions with human feedback
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• TinyBERT: Distilling BERT for Natural Language Understanding
• Patient Knowledge Distillation for BERT Model Compression
• Proximal Policy Optimization Algorithms
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Sequence-Level Knowledge Distillation
• Generative Adversarial Imitation Learning
• Distilling the Knowledge in a Neural Network
• Trust Region Policy Optimization
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Cited by

• OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

Related

• SODA: Semi On-Policy Black-Box Distillation for Large Language Models
• Lion: Adversarial Distillation of Proprietary Large Language Models
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Knowledge Distillation of Black-Box Large Language Models
• DDK: Distilling Domain Knowledge for Efficient Large Language Models