Running AI on mixed hardware for speed and affordability
| Source: IBM Research
Tags: llm-d, inference optimization, IBM Research, heterogeneous GPU, vLLM, Granite, sovereign cloud, SGLang
IBM Research, Red Hat, and NxtGen Cloud demonstrated that llm-d—an open-source inference orchestration layer for vLLM and SGLang—runs IBM Granite and Sarvam AI models on heterogeneous GPU clusters 3-5x faster with potentially 2x throughput versus unorchestrated serving.
Details
The research addresses a practical enterprise problem: most organizations deploying AI on-premise own a heterogeneous mix of GPU hardware across different generations and vendors, making efficient inference orchestration difficult. IBM Research, Red Hat, and NxtGen Cloud Technologies (one of India's sovereign cloud providers) ran experiments on NxtGen's production infrastructure using llm-d, an open-source Kubernetes-native system for orchestrating vLLM and SGLang inference engines. Two innovations drive llm-d's performance gains: a cache-aware router that directs incoming requests to the vLLM instance most likely to hold relevant KV cache data (reducing redundant recomputation), and prefill/decode disaggregation that separates prompt processing from token generation so each phase can be assigned to the most appropriate hardware tier. The result on NxtGen's multi-vendor GPU cluster: 3-5x faster inference and potentially 2x throughput for IBM Granite and Sarvam AI models compared to serving without llm-d. IBM calls this potentially the first documented demonstration of llm-d on multi-vendor GPU clusters in production conditions. The research was conducted in India, but IBM expects results to generalize to any open-source large-scale deployment. For enterprises with on-premise GPU fleets of mixed vintages, llm-d offers a path to significantly better utilization without hardware refresh.