ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration

| Source: arXiv AI

Tags: ROCm InferaSim, AMD Instinct MI355X, vLLM, SGLang, Prefill-Decode Disaggregation, KV Cache

ROCm InferaSim predicts LLM serving performance from a few short kernel benchmarks, replacing roughly 25,000 GPU-hours of trial-and-error tuning with single-digit GPU-hours and errors under 10%.

Details

Choosing an LLM serving setup means juggling engine, GPU split, precision, scheduler, speculative decoding, KV-cache handling and prefill-decode disaggregation. The authors estimate exhaustive measurement of their candidates would cost about 25,000 GPU-hours. ROCm InferaSim benchmarks each distinct kernel set once on at most one node (an anchor), then uses analytical laws to extrapolate to other batch sizes, prompt lengths and GPU counts. Multi-node setups are reduced to a single-node equivalent. A simulator replays multi-turn traffic and a tuning agent searches configurations. On AMD Instinct MI355X with vLLM, SGLang and ATOM, 11 anchors took 9.1 GPU-hours versus 204 GPU-hours to measure the 226 configurations they cover. Time per output token had 8.1% median error across 172 configs, and throughput for 159 disaggregated deployments was within 5.5%.