Embarrassingly Simple Self-Distillation Improves Code Generation

2026/04/01 by Ruixiang Zhang, Richard He Bai, Huangjie Zheng +3 · 36 voices · 2 citations
#cs.CL

paper · pdf

Abstract

Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation. Our code is available at https://github.com/apple/ml-ssd

Citations

• Self-Distillation Enables Continual Learning
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Competitive Programming with Large Reasoning Models
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Self-Rewarding Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
• Affective Coherence Monitoring for Transformer-Based Language Models
• Program Synthesis with Large Language Models
• Distilling the Knowledge in a Neural Network

Cited by

• LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
• Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

Discussions

• Embarrassingly simple self-distillation improves code generation [hn, 658 points, 201 comments]
• Embarrassingly Simple Self-Distillation Improves Code Generation [lobsters, 8 points, 4 comments]
• Especially funny when stuff like this exists: arxiv.org/pdf/2604.011... [bsky, 4 points, 0 comments]
• Embarrassingly Simple Self-Distillation Improves Code Generation [hn, 4 points, 3 comments]
• Apple: Embarrassingly Simple Self-Distillation Improves Code Generation https://arxiv.org/abs/2604.01193 https://news.ycombinator.com/item?id=47637757 [bsky, 2 points, 0 comments]
• Simple self-distillation helps AI write better code. Basically it gets better by just practicing, without any external feedback. [bsky, 1 points, 0 comments]
• Apple: Embarrassingly Simple Self-Distillation Improves Code Generation https://arxiv.org/abs/2604.01193 (https://news.ycombinator.com/item?id=47637757) [bsky, 1 points, 0 comments]
• Exciting advancements in code generation! Simple self-distillation techniques are enhancing model performance, demonstrating the power of iterative learning. Discover how these innovations are shaping [bsky, 1 points, 0 comments]
• Look, I don't know what this is, but I aspire to publish a research paper with the words "embarrassingly simple" in the title. arxiv.org/abs/2604.01193 [bsky, 1 points, 0 comments]
• Simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 https://news.ycombinator.com/item?id=47637757 [bsky, 0 points, 0 comments]
• Simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 comments #arxiv.org [bsky, 0 points, 0 comments]
• Apple: Embarrassingly Simple Self-Distillation Improves Code Generation https://arxiv.org/abs/2604.01193 [bsky, 0 points, 0 comments]
• Simple self-distillation improves code generation View Article | Join the HN Conversation Summary of HN discussion 🧵👇 [bsky, 0 points, 1 comments]
• Embarrassingly simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 [comments] [572 points] [bsky, 0 points, 0 comments]
• 📰 Embarrassingly simple self-distillation improves code generation 🔗 https://arxiv.org/abs/2604.01193 💬 Discuss on HN [bsky, 0 points, 0 comments]
• Apple shows that code generation can be great improved via distillation arxiv.org/abs/2604.01193 [bsky, 0 points, 0 comments]
• 🌐驚くほど単純な自己蒸留によりコード生成が改善される https://arxiv.org/abs/2604.01193 via #HackerNews [bsky, 0 points, 0 comments]
• Embarrassingly Simple Self-Distillation Improves Code Generation https://lobste.rs/s/bor4wy #ai [bsky, 0 points, 0 comments]
• Embarrassingly simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3miok36bsfgh2 [bsky, 0 points, 0 comments]
• Simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 (http://news.ycombinator.com/item?id=47637757) [bsky, 0 points, 0 comments]
• Simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 (http://news.ycombinator.com/item?id=47637757) [bsky, 0 points, 0 comments]
• Embarrassingly simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 (http://news.ycombinator.com/item?id=47637757) [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2604.01193 この研究は、自己蒸留を用いてコード生成の性能を向上させます。 特に「embarrassingly simple」とあるように、その手法は非常にシンプルです。 シンプルなアプローチにもかかわらず、高い効果を発揮することが示されています。 [bsky, 0 points, 0 comments]
• Apple: Embarrassingly Simple Self-Distillation Improves Code Generation view on hacker news [bsky, 0 points, 0 comments]
• A recent study highlights Simple Self-Distillation (SSD), enabling large language models to enhance code generation with their outputs. SSD improves Qwen3-30B-Instruct model accuracy from 42.4% to 55. [bsky, 0 points, 0 comments]
• Embarrassingly simple self-distillation improves code generation https:// arxiv.org/abs/2604.01193 # arxiv [mastodon, 0 points, 0 comments]
• Embarrassingly Simple Self-Distillation Improves Code Generation arxiv.org/abs/2604.01193 [bsky, 0 points, 0 comments]
• Simple self-distillation improves code generation https://arxiv.org/abs/2604.01193 (https://news.ycombinator.com/item?id=47637757) [bsky, 0 points, 0 comments]
• Apple: Embarrassingly Simple Self-Distillation Improves Code Generation https://arxiv.org/abs/2604.01193 (https://news.ycombinator.com/item?id=47637757) [bsky, 0 points, 0 comments]
• [2604.01193] Embarrassingly Simple Self-Distillation Improves Code Generation — A clean, practical paper on using simple self-distillation to boost code generation quality. Worth a skim if you’re tuni [bsky, 0 points, 0 comments]
• Embarrassingly Simple Self-Distillation Improves Code Generation arxiv.org/pdf/2604.01193 [bsky, 0 points, 0 comments]
• 1/2 Simple auto-distilado mejora generación de código Un paper de arXiv demuestra que aplicar auto-distilación simple (el modelo aprende de sus propias predicciones) mejora la generación de código sin [bsky, 0 points, 1 comments]
• Apple's Simple Self-Distillation enhances LLM code generation, improving accuracy and simplifying training without external verification or teacher models through embarrassingly simple supervised lear [bsky, 0 points, 0 comments]
• Simple self-distillation improves code generation [bsky, 0 points, 0 comments]
• Apple: Embarrassingly Simple Self-Distillation Improves Code Generation #HackerNews https://arxiv.org/abs/2604.01193 [bsky, 0 points, 0 comments]

Related

• Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
• SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs
• Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models
• Dynamic Corrective Self-Distillation for Better Fine-Tuning of Pretrained Models
• SelfCodeAlign: Self-Alignment for Code Generation