Embarrassingly Simple Self-Distillation Improves Code Generation

July 16, 2026 | Source: Apple ML Research

Tags: Apple ML Research, self-distillation, code generation, Qwen3, Llama, LiveCodeBench, SFT, post-training

Apple researchers show LLMs can boost code generation by ~30% relative using only their own sampled outputs — no teacher model, no RL, just temperature-varied SFT — lifting Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6 and challenging assumptions that external feedback is required for self-improvement.

Details

Apple ML Research published a paper showing that 'simple self-distillation' (SSD) — sampling a model's own outputs at varied temperatures and truncation settings, then fine-tuning on those samples with standard supervised fine-tuning — produces meaningful code generation gains without any external signal source. The headline result: Qwen3-30B-Instruct improves from 42.4% to 55.3% pass@1 on LiveCodeBench v6, a roughly 30% relative jump. Gains concentrate on harder problems. The method generalizes across Qwen and Llama families at 4B, 8B, and 30B scale, covering both instruct and thinking model variants. The paper attributes the improvement to a 'precision-exploration conflict' in LLM decoding. SSD reshapes token distributions in a context-dependent way: suppressing 'distractor tails' where exact precision matters while preserving output diversity where exploration helps. This framing connects SSD to distillation theory without requiring a separate teacher model. The practical significance is high for teams iterating on code models: SSD is a cheap, low-complexity post-training step with no external dependency. Apple positions it as complementary to RL and preference learning rather than a replacement. The result on thinking-mode variants is particularly relevant given the current shift toward reasoning models.