Presentation: Producing the World's Cheapest Tokens: A How-to Guide
| Source: InfoQ AI/ML
Tags: LLM inference, cost optimization, speculative decoding, batch processing, Doubleword, QCon
Doubleword CEO Meryem Arik lays out a framework for cutting LLM inference costs 2–10x for high-volume, non-real-time workloads by optimally matching hardware selection, batch sizes, speculative decoding, and request scheduling to the actual use case.
Details
Most enterprises are paying 2–5x more than necessary for LLM inference because their stack doesn't match their workload priorities, argues Meryem Arik, CEO of self-hosted inference platform Doubleword. In her QCon San Francisco presentation, she frames inference as a three-way trade-off between latency, cost, and quality — and argues that non-real-time, high-volume use cases can aggressively sacrifice latency to cut costs by an order of magnitude.\n\nThe core strategy involves four levers: hardware selection (choosing the right accelerators rather than defaulting to highest-spec GPUs), batch size optimization (packing more requests together to maximize throughput), speculative decoding (small draft model generates candidates, large verifier accepts or rejects), and smart queue reordering (scheduling requests to minimize idle compute time).\n\nArik frames this as an engineering discipline most teams haven't treated rigorously. For high-volume jobs that don't need real-time responses — document processing, batch analytics, offline RAG pipelines — the economics look completely different from interactive chat.