BenchMIRT: What are LLM benchmarks actually measuring?

Details

LLM benchmark scores are routinely used to rank and select models, but Ai2 (Allen Institute for AI) argues the single aggregate scores hide as much as they reveal. BenchMIRT, published today with code and data, applies multidimensional IRT — a psychometrics technique — to decompose benchmark scores into their underlying capability components. The system was trained on results from 100 LLMs evaluated across 16 benchmarks covering more than 34,000 individual questions. Six benchmarks target general reasoning (including MMLU-Pro, GPQA, MATH, and BBH); the remaining 10 cover safety, instruction-following, and related abilities. Two concrete examples illustrate the problem BenchMIRT addresses. BBQ, designed to test social-stereotype bias, also requires multi-hop reasoning — so a model's score reflects both bias tendencies and reasoning ability, not just one. WildJailbreak mixes harmful jailbreak prompts (safety signal) with benign refusal prompts (general reasoning signal); averaging them masks how a model performs on each type. BenchMIRT lets researchers separate those signals and identify which capability is actually driving a score on any given question. Practically, this matters when choosing a model for a specific use case: a high safety benchmark score might be earned mostly through strong reasoning rather than robust safety alignment. BenchMIRT gives benchmark designers and model evaluators a diagnostic tool to catch those confounds — and to design cleaner benchmarks going forward.