CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?

Details

Video world models are increasingly framed as physics simulators, but existing benchmarks score individual generation quality or compare distributions coarsely — without testing whether models reproduce the correct probability over physically outcomes. CaliBench addresses this by scoring outputs in discrete, physically interpretable outcome spaces (bin index, die face, card suit, roulette color) where the reference distribution is known in closed form.\n\nThis design enables an exact calibration test using a chi-squared test against known reference distributions (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, skewed European roulette). Performance is decomposed into scorability (fraction of generations yielding a scoreable outcome) and calibration (total variation distance from the reference on that sample).\n\nSix image-to-video models were evaluated on nine scenes with 32 generations each: WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, and Cosmos3-Super. Most scene-model combinations are significantly miscalibrated — models concentrate probability mass on a few outcomes rather than reproducing the reference distribution. The extreme case: Veo 3.1 collapses to a single die face on dice scenes. On roulette, several models have low scorability as the ball lands ambiguously. No model dominates all nine scenes. The paper releases the mnTV (mean normalised total variation) metric and protocol.