2026/06/26 by Jiangan Yuan, Zhixuan Li, Han Xu
#cs.LG #cs.AI
Off-policy distillation is now central to large language model pre-training, yet how training data, objective parameterization, and model capabilities interact remains poorly characterized. We studies top-k-truncated, temperature-scaled off-policy distillation by decomposing this problem into two questions: an objective-to-capability analysis of how the training objective shapes token-level supervision and downstream performance, and a data-to-objective analysis of how data heterogeneity should inform objective routing. We first show that the language-modeling objective (LLM) and the knowledge-distillation objective (LKD) induce systematically different capability profiles, and trace this divergence to a gradient-level tension between direct observed-token reinforcement and teacher-supported alternative supervision. To quantify this tension, we introduce diagnostic metrics -- support coverage, observed-token probability mass, and teacher-distribution concentration -- and show via controlled sweeps that the support size k governs a coverage-sharpness trade-off, while distillation temperature controls within-support probability allocation. We then examine adaptive objective routing: a domain-level policy that applies LLM to math and code and LKD to general-domain data yields consistent gains over both single-objective baselines, whereas token-level routing based on observed-token probability mass or teacher entropy fails to consistently match the single-objective baseline. These results suggest that effective objective routing depends less on routing granularity than on the quality of the routing signal, reframing continued pre-training via off-policy distillation as a structured, data-conditional supervision-design problem rather than a global hyperparameter choice.