Latest open artifacts (#21): Open model bonanza! Gemma 4, DeepSeek V4, Kimi K2.6, MiMo 2.5, GLM-5.1 & others. On CAISI's V4 assessment.

Details

May 2026 saw an unusual concentration of open-frontier model releases in a single month. Standout from the batch: Xiaomi's MiMo-V2.5-Pro (Apache 2.0 licensed), which the authors describe as competitive with Kimi K2.6 and GLM-5.1 in both benchmarks and real-world use — notable given Xiaomi launched its first model only one year prior. The US Center for AI Standards and Innovation (CAISI) published a DeepSeek V4 evaluation using nine benchmarks and an IRT-based Elo scoring system. Their headline finding: open models trail the closed American frontier, with the gap growing over time. The Elo differential is largely driven by DeepSeek V4's performance on CTF-Archive-Diamond, the CAISI-private PortBench, and ARC-AGI-2 scored with a non-standard method — all of which skew the aggregate result significantly. Interconnects' authors push back: standard benchmark setups (basic bash access with fixed token budgets) don't reflect how these models are actually deployed in practice, e.g., inside coding harnesses like Claude Code. Epoch AI's ECI metric places open models 3–7 months behind closed frontier since R1 — a more conservative and arguably more defensible gap estimate. The broader debate over benchmark methodology vs. real-world capability remains unresolved. Other releases in the batch include Gemma 4 (26B MoE, 4B active parameters, covered separately) from Google. The article is partly paywalled; not all release details were available in the scraped excerpt.