Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation
| Source: arXiv AI
Tags: knowledge distillation, label smoothing, Bangla, low-resource NLP, EWAD, CHAD
A study of gated knowledge distillation for Bangla generation finds no gate beats plain distillation in 45 comparisons, and that distillation's main benefit is regularization that label smoothing largely matches.
Details
Gated knowledge distillation tries to trust an imperfect teacher only where it looks reliable. The authors test two gates, EWAD (token-level, by teacher confidence) and CHAD (example-level, by gradient alignment with validation data). Across six Bangla-centric tasks, three students from 60M to about 2B parameters, and two teacher families, no gate significantly improved plain KD in 45 comparisons. KD also failed to beat cross-entropy fine-tuning on gold outputs, and CE led on every summarisation metric. The main KD effect was regularisation: over ten epochs CE students lost 0.030 to 0.044 token-F1 from their best checkpoint versus 0.005 to 0.025 for distilled students, and teacher-free label smoothing closed most of that gap. Findings are limited to Bangla tasks and small students.