GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix
| Source: arXiv AI
Tags: KV-cache, LLM serving, multi-agent systems, inference optimization, Llama, Qwen, throughput
GraniKV separates KV-cache storage into a contiguous pool for shared LLM prefixes and a token-level pool for per-request suffixes, delivering 2.16× output throughput over production baselines for multi-agent workloads with 16K shared prefix tokens on Llama-3.1-8B.
Details
Production LLM serving engines apply uniform KV-cache paging granularity to all requests, even though multi-agent workloads have fundamentally different storage requirements in two regions: a long shared prefix (same system prompt or document across all agents) demands contiguous memory, while the per-request suffix demands fine-grained allocation. GraniKV identifies and exploits this asymmetry. The system allocates shared prefixes in a contiguous HOT pool and suffixes in a token-level COLD pool, with a per-step dispatcher routing compute across dual backends — selecting between compute-, memory-, or communication-bound processing for each regime. The asymmetric storage design enables batched-GEMM computation on shared prefixes that uniform paging cannot support. Benchmarks at 16K shared tokens show 2.16× output-token throughput over the production baseline on Llama-3.1-8B (TP=1), 1.98× on Qwen-2.5-14B (TP=2), and 1.57× on Qwen-2.5-32B (TP=4). Under heterogeneous multi-agent serving with prompts of different lengths — the hardest real-world case — GraniKV holds 1.95× throughput while batch-global cascade attention collapses to parity. The asymmetric storage layer alone contributes 1.05–1.15× end-to-end. Source is an arXiv preprint.