GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts
| Source: arXiv AI
Tags: GC-MoE, Mixture of Experts, CUDA Green Contexts, DeepGEMM, DeepSeek-V4.1-Flash, GLM-5.2, Blackwell, FP4
GC-MoE uses CUDA Green Contexts to split GPUs into separate communication and compute partitions for MoE inference, speeding DeepSeek-V4.1-Flash decode by about 1.19x on 8 H20s and beating DeepGEMM Mega MoE.
Details
Under expert parallelism, MoE decode is dominated by all-to-all dispatch latency and weight traffic. Fused operators like DeepGEMM's Mega MoE overlap dispatch and compute by sharing every SM, so the phases compete for registers, shared memory and issue slots. GC-MoE is an open-source FP8/FP4 MoE operator for Hopper and Blackwell that uses CUDA Green Contexts to give dispatch and grouped GEMM their own partitions, kernels and sync domains. A small CUDA Graph launches both concurrently, a token-ring handshake starts compute early, and double-buffered counters remove the trailing cross-rank barrier. On 8 H20 GPUs it speeds DeepSeek-V4.1-Flash decode by 1.188x on average (peaks 1.296x and 1.353x). On 8 B300 GPUs it speeds GLM-5.2 and DeepSeek-V4.1-Flash by 1.106x, and beats Mega MoE in all reported measurements.