Mixture-of-Expert Blocks Contain Strong Hallucination Detection Signals

Details

Hallucination detection methods mostly operate at the answer or sentence level. Per-token detection is harder but necessary for localizing exactly which spans are hallucinated and enabling surgical interventions. InnerExpert is the first method to exploit MoE-specific internal signals for this task. In MoE architectures, each forward pass activates a sparse expert subset via routing. This produces signals invisible in dense models: router entropy (how spread the load is), expert disagreement (whether different experts would route differently), and expert usage patterns (which experts activate together). InnerExpert combines these routing signals with standard transformer features into compact per-token feature vectors, then trains a lightweight detector using labels from an LLM-as-a-judge pipeline — enabling continuous retraining without manual annotation. Results across five datasets and two MoE architectures: 0.91 answer-level AUROC and 0.76 token-level AUROC. The approach outperforms existing detection methods on both metrics and runs on a single forward pass. The practical implication: teams deploying MoE models (Mixtral, DeepSeek MoE, similar architectures) can attach lightweight per-token hallucination detection at marginal additional cost relative to dense-model alternatives.