Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

2026/06/30 by Guo Yu, Wenlin Liu, Yulan Hu +3
Computer Science · #cs.LG

paper · pdf

Code is available at https://github.com/SydCS/OPD-Param-Analysis

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

On-policy distillation (OPD) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student-generated trajectories and dense token-level teacher supervision. Yet how this hybrid training regime shapes a model remains poorly understood. We characterize the sparsity and geometry of OPD parameter updates across several language and vision-language model pairs and application settings. OPD updates are small and coordinate-sparse at checkpoint precision, while remaining distributed across layers and modules. This sparse support is operationally meaningful: masked training on the discovered subnetwork nearly recovers full-training performance. At the matrix level, the updates are numerically full-rank but spectrally concentrated. Their visible supports avoid coordinates emphasized by the source's principal structure and favor low-magnitude source coordinates, while the source singular-value spectra change little. Together, these findings show that OPD exhibits important weight-space signatures of on-policy post-training despite using dense teacher supervision.

Citations

• GLM-5: from Vibe Coding to Agentic Engineering
• Reinforcement Learning via Self-Distillation
• JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
• The Path Not Taken: RLVR Provably Learns Off the Principals
• On Predictability of Reinforcement Learning Dynamics for Large Language Models
• RL's Razor: Why Online Reinforcement Learning Forgets Less
• OpenThoughts: Data Recipes for Reasoning Models
• Who Reasons in the Large Language Models?
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
• Qwen3 Technical Report
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Muon is Scalable for LLM Training
• Qwen2.5-VL Technical Report
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Editing Models with Task Arithmetic
• LoRA: Low-Rank Adaptation of Large Language Models
• Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Sequence-Level Knowledge Distillation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Distilling the Knowledge in a Neural Network
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning

Related

• RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models
• Let the Data Decide: Supervision Analysis, Capability Trade-offs, and Adaptive Objective Routing in Continued Pre-Training via Off-Policy Distillation
• SODA: Semi On-Policy Black-Box Distillation for Large Language Models
• Weak-to-Strong On-Policy Distillation
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation