What We Learned by Reproducing 2,200 papers from ICML

Details

ICML 2026 received 23,918 submissions and accepted 6,352 papers — roughly double the previous year, a trend partly driven by AI agents accelerating experiment-running and paper writing. Reviewing capacity hasn't kept pace: a volunteer reviewer of one accepted spotlight paper explicitly wrote 'I did not check all the proofs carefully' despite the paper receiving strong scores.\n\nFrom July 15 to August 2, Hugging Face ran the ICML 2026 Open Reproductions challenge. 1,200+ participants brought their own coding agents and worked claim by claim through accepted papers. Agents used included Claude Code, Codex, Cursor, and OpenResearch's orx. In 19 days, participants published 6,816 Trackio logbooks reproducing 2,226 papers — approximately a third of the conference. The same spotlight paper with unchecked proofs was one of the papers finally reviewed carefully — and errors were found.\n\nThe challenge infrastructure provided each participant with extracted core claims from every paper so agents could start from a concrete, checkable target rather than a 40-page PDF. Each reproduction was logged publicly. The Hugging Face team frames the central question as: if AI is accelerating submissions exponentially, can it also serve as a reproducibility counterweight at the same scale?\n\nThe broader structural implication is significant: the bottleneck in AI research has shifted from compute to credibility infrastructure. Peer review designed for human throughput cannot scale with AI-generated submissions. Agent-driven reproducibility audits may become a necessary quality mechanism for major ML conferences.