How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation

Details

LinkedIn published a detailed engineering breakdown of how it built and trained the ranking model behind its AI-powered job search. The core product is a 0.6B-parameter student model trained by distilling outputs from larger teacher LLMs — but at LinkedIn's scale (hundreds of thousands of queries per second), serving those teachers during training was itself a serious bottleneck. The team built two complementary distillation modes on top of SGLang: Online Multi-teacher Distillation asynchronously queries teacher models during training and, with local replicas spread across nodes, delivers a 3x speedup; Offline Multi-teacher Distillation precomputes teacher outputs to HDFS or NFS so they can be reused directly, eliminating repeated inference entirely. Both modes support multiple teacher sizes via tensor-parallel and data-parallel configuration. On top of the distillation framework, LinkedIn stacked a series of training-level optimizations: LiGer for 2x larger batch sizes through reduced memory use, multi-node training for up to 3.5x additional speedup, FSDP2 for a 20% gain, and H200 multi-node clusters for a further 30%. One notable negative result: FP8 mixed precision provided no benefit for models under 8B parameters, as casting overhead outweighed any compute savings. Combined, these techniques deliver roughly 8x faster training. The write-up is most useful for ML teams currently running keyword-based search or recommendation stacks and looking to move to LLM-supervised unified rankers without grinding iteration cycles to a halt.