S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

Details

Deploying MoE models on edge devices faces two compounding problems: memory constraints limit which experts can be resident at once, and speculative decoding—the standard inference acceleration technique—becomes inefficient on MoE architectures because verifying draft tokens requires re-routing through experts that may not be cached. S2-MoE addresses this with three coordinated mechanisms. Routing-aware adaptive speculative expansion selects draft lengths based on predicted routing consistency, avoiding expensive verification for tokens likely to diverge. Reuse-aware expert gating prioritizes verification of tokens whose required experts are already loaded, minimizing memory transfers. Shared context alignment ensures draft and target model execution use the same cached state, reducing redundant computation. Evaluated across diverse MoE models and datasets on edge hardware (implementation in MLC LLM), S2-MoE achieves up to 5.3x speedup over standard autoregressive decoding, with approximately 2x average across the full test suite. No quality degradation is reported. The code is publicly available. For practitioners targeting on-device deployment of MoE models—particularly Mixtral-class or similar architectures—S2-MoE offers a meaningful inference acceleration path without requiring hardware upgrades or model quantization tradeoffs.