Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation

August 18, 2026 | Source: arXiv AI

Tags: LLM evaluation, LLM judges, rubric evaluation, self-distillation, Qwen3, Llama-3.1, HealthBench

SARA (Self-Anchored Rubric Alignment) fixes rubric interference in LLM judges—where multi-rubric batch evaluation produces inconsistent verdicts depending on which criteria co-appear—using on-policy self-distillation with no external supervision required.

Details

LLM-as-judge systems increasingly evaluate outputs against multi-criterion rubric checklists. Two approaches exist: separate inference calls per criterion (accurate, expensive) or batch evaluation in one pass (cheaper, but inconsistent). This paper quantifies just how inconsistent batch evaluation is and proposes a self-supervised fix. The authors measure rubric interference by probing four controlled operations: rubric set expansion, subsetting, reordering, and noise injection. In a preliminary study, only one-third of samples receive fully consistent verdicts when rubric set composition varies—a significant reliability problem for anyone using batch multi-rubric LLM evaluation in production. SARA addresses this without labeled data: it uses the model's own single-rubric judgments as stable anchors and aligns multi-rubric reasoning with these anchors through on-policy self-distillation. Evaluated on HealthBench, FLASK, and ResearchQA using Qwen3 and Llama-3.1, SARA consistently improves evaluation consistency while maintaining agreement with GPT-4.1 as a reference judge. The consistency improvement transfers across datasets, confirming SARA teaches a general capability rather than dataset-specific patterns. For production teams using LLM judges for multi-criteria evaluation, the implication is clear: your current batch rubric scores are likely less consistent than single-criterion evaluation would suggest.