HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models
| Source: arXiv AI
Tags: HybridSpec, speculative decoding, Gated DeltaNet, mixture-of-experts, DFlash, Qwen3.6-35B-A3B
HybridSpec picks speculative-decoding verification length on the GPU for hybrid Gated DeltaNet and MoE models, running 1.75x-1.88x faster than autoregressive decoding and about 1.3x faster than DFlash.
Details
In speculative decoding with parallel drafters such as DFlash, target verification dominates each round. For hybrid models mixing Gated DeltaNet (GDN) and mixture-of-experts layers, verification cost climbs steeply with length: on Qwen3.6-35B-A3B it rose 71% from 4 to 16 tokens, versus 4-6% for dense Qwen3-4B. HybridSpec chooses the verification length on the GPU from draft acceptance estimates and measured latency, then runs one extra token up to the first MoE router and uses observed expert routing to decide whether to keep it. Block-parallel GDN kernels rebuild accepted recurrent states in one launch and run 1.54x-4.04x faster than default kernels in isolated tests. Across three GDN-MoE targets in single-request greedy decoding, HybridSpec is 1.75x-1.88x faster than autoregressive decoding and 1.30x-1.33x faster than DFlash.