FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction
| Source: arXiv AI
Tags: Mixture of Experts, MoE, inference optimization, distributed inference, load balancing, LLM serving
FreeBalance cuts MoE distributed inference latency by 13.1% and reduces expert load imbalance by 32.8% by predicting routing distributions before they're needed — letting expert migration overlap with preceding computation instead of sitting on the critical path.
Details
In Mixture-of-Experts (MoE) distributed inference, load imbalance is a persistent bottleneck: when routing assigns tokens unevenly across experts, the most-loaded rank stalls the entire pipeline. Offline expert placement helps but cannot handle real-time workload dynamics across tasks, layers, and batch compositions. FreeBalance attacks the root problem: existing online balancing waits for routing decisions before migrating experts, placing migration latency squarely on the critical path. FreeBalance instead predicts the routing distribution before routing happens — using cross-layer similarities in hidden residual representations to build a lightweight workload predictor. With predictions available before routing, expert migration can be overlapped with preceding computation (e.g., attention). A cost model constrains the number of swaps to ensure synchronization overhead is fully hidden within the available window. Across models and datasets, FreeBalance reduces the max-to-mean load ratio by 32.8% and end-to-end prefill latency by 13.1%, hiding an average of 5.1 expert migrations per layer (which would otherwise cost ~8.5% of critical-path latency). This is particularly relevant for production deployments of large MoE models like Mixtral, DeepSeek-V3, or similar. The 13.1% prefill latency reduction on top of already-deployed MoE inference infrastructure is a meaningful practical gain.