Look Before You Launch: Learned Throughput Prediction for Hardware Selection in a Production Training Fleet
| Source: arXiv AI
Tags: LEAP, throughput prediction, mixed-vendor accelerators, GPU scheduling, ML infrastructure
LEAP, a production service that predicts training-job throughput across 20+ accelerator types, now ranks 95% of eligible GPU-hours and saves an estimated 3% of training compute.
Details
In mixed-vendor accelerator fleets, routing a training job well requires knowing its throughput on each hardware type before it runs. LEAP predicts each job's throughput on every candidate type at submission time and ranks options by throughput per unit cost. It routes tens of thousands of job attempts daily across hundreds of model types and more than 20 hardware types. It reaches 77% accuracy over 30 nightly retrains, beating the lookup table it replaced and every alternative tested, including fine-tuned language models. Over six months, ranked GPU-hours grew from 58% to 95%, with an estimated 3% saving in training compute. The paper also lists which factors actually moved accuracy.