Creating Power Flexibility in Datacenters via Query Routing
| Source: arXiv AI
Tags: H100, query routing, datacenter power, demand response, LLM inference, grid flexibility
A query-routing framework lets LLM datacenters cut power draw on demand without deferring requests, reaching up to 16% power flexibility in a simulated 10,000-H100 site, with 90% of the target cut landing within about 10 seconds.
Details
LLM inference is pushing datacenter power demand up, and grid operators increasingly want large loads that can flex. This paper looks at query routing, shifting requests across different model instances, as a way to change power use without delaying interactive traffic. The authors build an analytical framework linking query arrival rates and sequence-length distributions to token iteration time, residence time, serving capacity and power, then use it to pick routing strategies that hit a target power level. Validated on four GPU platforms with models from 1B to 70B parameters, the framework estimates achievable power flexibility with 2.8% mean absolute percentage error. In a trace-driven simulation of a 10,000 H100 GPU datacenter, routing delivered up to 16% flexibility of total power, with 90% of the requested reduction reached in roughly 10 seconds.