Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs

| Source: arXiv AI

Tags: vLLM, SGLang, Quantization, Gemma 3, Qwen2.5, FP8, AWQ

An audit of public benchmarks for serving 9B-20B open LLMs finds 4-bit quantization roughly doubles small-batch speed but the gain disappears at large batches, and many reported accuracy losses sit within noise.

Details

Teams serving mid-size open-weight models on vLLM or SGLang need two numbers for each quantization recipe: speedup over BF16 and accuracy lost. The authors collected every public measurement they could verify and found the record fragmentary. Only two checkpoints, both INT4 builds of Gemma 3 12B, had both a vLLM timing and an accuracy score, and that score covered one MMLU subject the AWQ build was calibrated on. AMD and Intel Gaudi had no timings in this band, and FP4 had none against BF16. The timings follow one rule: small-batch decoding is memory-bound, so 4-bit weights give about 2x (2.12x for Qwen2.5-14B on an A100), but at batch 256 the GPU is compute-bound and the gain drops to 1.02x for Gemma 3 12B, where FP8 can still help. The paper ends with a seven-item reporting checklist.