Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
| Source: Apple ML Research
Tags: speculative decoding, inference efficiency, reasoning, Apple, UC Berkeley, LLM inference
ARBITRAGE, from Apple and UC Berkeley researchers, cuts LLM reasoning inference latency by up to 2x by dynamically routing generation to a fast draft model or powerful target model step-by-step — avoiding the token-mismatch failures that make standard speculative decoding ineffective on reasoning tasks.
Details
Standard speculative decoding speeds up LLM inference by having a small draft model propose tokens that a larger model verifies in parallel — but on reasoning tasks, token-level mismatches between draft and target cause excessive rejections even when the underlying reasoning is semantically equivalent, limiting the speedup. ARBITRAGE addresses this by operating at the step level rather than the token level. A lightweight router is trained to predict when the target model will produce a meaningfully better reasoning step than the draft model, enabling dynamic routing rather than a fixed acceptance threshold. This approximates an ideal ARBITRAGE ORACLE that always selects the higher-quality step. Across multiple mathematical reasoning benchmarks, ARBITRAGE reduces inference latency by up to approximately 2x at matched accuracy, consistently outperforming prior step-level speculative decoding baselines. The router itself is lightweight, meaning routing overhead doesn't cancel out the speedup. For teams running expensive reasoning models in production, a 2x latency reduction at no accuracy cost is an engineering result worth evaluating.