DOPD: Dual On-policy Distillation
2026/06/29 by Xinlei Yu, Gen Li, Qingyi Si +13 · 1 voice
Computer Science · #cs.AI
paper · pdf
Abstract
On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.
Citations
- JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
- The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
- Reinforcement Learning via Self-Distillation
- Self-Distillation Enables Continual Learning
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
- Qwen3-VL Technical Report
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Skywork Open Reasoner 1 Technical Report
- LaViDa: A Large Diffusion Language Model for Multimodal Understanding
- Qwen3 Technical Report
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
- AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
- Small Models Struggle to Learn from Strong Reasoners
- Distillation Scaling Laws
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
- DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
- LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- BioXP-0.5B: Explainable Medical-AI via RL-GRPO
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
- Measuring Mathematical Problem Solving With the MATH Dataset
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Sequence-Level Knowledge Distillation
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
Discussions
- [29/30] 209 Upvotes, 2 Comments, 2 Posts, arXiv:2606.30626 🆕DOPD: Dual On-policy Distillation Xinlei Yu, Gen Li, Qingyi Si, Guibin Zhang, Yuqi Xu, Congcong Wang [bsky, 0 points, 1 comments]
Related