Shuffled OPSD: When Semantic Consistency Hurts On-Policy Self-Distillation?
| Source: arXiv AI
Tags: On-Policy Self-Distillation, Qwen3, Knowledge Distillation, Mathematical Reasoning, Shuffled OPSD
Shuffling the words of the shared problem statement in on-policy self-distillation improved math reasoning accuracy across three Qwen3 scales, with up to 3.52 points over standard OPSD on Qwen3-1.7B.
Details
On-policy self-distillation (OPSD) uses one model as both teacher and student: the student reasons from the question alone, while the teacher also sees a worked example and gives token-level supervision. Shuffled OPSD scrambles the word order of the shared problem statement while leaving the teacher-only worked example intact, testing whether preserving semantic structure actually helps learning. Across three math reasoning benchmarks and three Qwen3 sizes, the method beat both an intact-target control and standard OPSD under matched training settings. The largest gain over standard OPSD was 3.52 percentage points on Qwen3-1.7B. Controls showed that keeping larger word blocks gave no advantage over word-level shuffling at the tested scales, which matters for teams tuning distillation recipes for small reasoning models.