All of AI benchmarking at your fingertips

Details

AI benchmarking has a reproducibility problem: more than 24 different evaluation harnesses exist, each tallying scores differently, and results scatter across leaderboards, papers, and blog posts with no common format. Identical evaluations on the same model can produce scores diverging by up to 20 percentage points — a gap large enough to flip which model appears best. A documented case: Hugging Face researchers traced conflicting LLaMA MMLU scores back to different harnesses being used, not differences in the model itself.\n\nEveryEvalEver, led by IBM Research with Hugging Face and Technical University of Munich, attacks this at the infrastructure level. The project introduces a shared reporting language for benchmark results and a crowdsourced database hosted on Hugging Face, currently holding 22,000+ model results across 2,200 benchmarks — all translated from 31 evaluation formats into a unified schema.\n\nThe scientific motivation is direct: if benchmark scores guide model selection in production and research, those scores need to be comparable and reproducible. Leshem Choshen (IBM) noted that evaluation decisions are currently driven by gut feeling because the underlying data is too fragmented to reason about rigorously. The broader research community has echoed this in the paper 'Benchmarking is Broken'.\n\nFor practitioners, this means a single queryable source of truth for benchmark comparisons rather than manually hunting across disparate leaderboards. The database is live on Hugging Face and open for community contributions.