Making Knowledge Distillation Cheap Enough to Run at Scale
| Source: Hugging Face Blog
Tags: knowledge distillation, LLM compression, Multiverse Computing, VRAM optimization, KL divergence, open-source LLMs
Multiverse Computing's paper cuts LLM knowledge distillation costs by caching teacher model top-K logits offline and using a memory-efficient chunked KL loss — enabling long-context distillation on a single GPU instead of requiring hundreds of GPUs.
Details
Knowledge distillation — training a smaller student model to match a larger teacher — is expensive because both models must run simultaneously while processing each token's full vocabulary distribution. For a model like gpt-oss-120b with 201,088 vocabulary tokens, this creates tensors so large that large-scale distillation is normally only feasible with multi-GPU clusters.\n\nMultiverse Computing's paper tackles this with two changes. First, it caches the teacher's top-K logits offline in a single pass, so the teacher never needs to be loaded during student training. Second, it introduces a fused chunked KL-divergence loss that processes vocabulary in chunks instead of materializing the full vocabulary-size times sequence-length matrix, cutting peak VRAM far below what PyTorch or NVIDIA Megatron-Bridge produce by default.\n\nThe practical outcome: long-context healing — a training technique for recovering model quality at extended context lengths — is now feasible on a single GPU, and distillation experiments that previously required cluster access become cheap enough to run routinely. Multiverse Computing has applied these techniques in their Hypernova 60B release; Nvidia's Nemotron 3 Puzzle 75B is a comparable industry example of the same compression approach.