Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing

Details

Mixture-of-Experts models route each token to a subset of experts rather than activating all parameters — but what does that routing actually encode? This paper introduces J64 (expert-load entropy across the sequence) and R64 (routing variability across layers) as lightweight scalar readouts that track reasoning depth without rerunning the full model. Across four MoE reasoning models tested on math, coding, and science benchmarks, median per-axis Pearson correlations between J64/R64 and accuracy range from 0.69 to 0.86. The metrics are stable enough that routing-only proxies — computed at a fraction of full inference cost — retain 0.9 to 3.2 accuracy points versus the complete model, depending on task difficulty. The practical interpretation: J64 and R64 behave as difficulty meters. High routing entropy and variability signal the model is distributing work broadly, a pattern correlated with harder problems and correct answers. Low values on supposedly hard questions are a warning sign worth inspecting. For teams building MoE-based inference pipelines, these metrics open a path to early-exit, adaptive compute, and selective full-run decisions without waiting for final output generation.