Self-Supervised Visual On-Policy Distillation

Details

Improving vision-language models typically requires either a larger teacher model, ground-truth annotations, or reference answers — all forms of privileged information that raise cost and limit deployment. S²VOPD (Self-Supervised Visual On-Policy Distillation) achieves comparable gains without any of these. The core insight inverts the standard privileged-information framing: instead of giving the teacher more, the method subtracts information from the student. The teacher sees the original image; the student sees a strongly augmented version. This asymmetry creates a learning signal equivalent to having access to information the student lacks — purely from data augmentation. Three design findings shape the method: (1) augmentation asymmetry matters — all four families tested improve performance while symmetric self-distillation degrades it; (2) augmentation strength has an optimal point — too weak loses signal, too strong destroys task-relevant content; (3) augmentations must stay task-consistent — removing evidence needed to answer the question creates uninformative discrepancies. Evaluated on six fine-grained visual perception benchmarks, S²VOPD improves Qwen3.5-4B from 70.7% to 77.4%, surpassing all open-source models up to Qwen3-VL at 235B and exceeding GPT-5.4. It recovers 96% of the improvement achieved by methods with privileged information — at zero additional labeling cost.