An Omitted Mode Is a Rare Rule: The Sampling-Verification Danger Law in Continuous Code World Models
| Source: arXiv AI
Tags: AI safety, Code World Models, LLM planning, formal verification, GPT-5, autonomous systems
A 92-page formal analysis proves that LLM-synthesized Code World Models can pass rollout acceptance tests while hiding dangerous behavioral modes — GPT-5.x repairs simple 1D rule omissions 95% of the time but fails entirely on 2D regions across 156 artifacts, proving acceptance certifies sample consistency, not safety.
Details
The Code World Model paradigm uses LLMs to synthesize executable environment simulators for classical planners. This paper formalizes what rollout acceptance testing — validating the model against sampled transitions — actually certifies about rare-but-critical behaviors. The answer is stark. When a critical event has probability r per rollout, N acceptance samples miss it with probability (1-r)^N — the paper calls this the Sampling-Verification Danger Law. This is demonstrated on three hybrid control instruments where mode-blind accepted models are exploited by a planner pinned at mode boundaries, yielding near-full attainable return as regret. Empirically with GPT-5.x: the model perfectly repairs an omitted 1D clamp rule in 105 of 111 mode-containing draws (exact on 50/56 blocks, 95% CI [0.781, 0.960]). On 2D regions, 0 of 156 artifacts recover the rule after 8 targeted interventions. A localization budget theorem gives formal bounds: models with Lipschitz constant L disagreeing by η at a point diverge above tolerance ε on a region of volume at least κ((η-ε)/L)^(d+m). For AI safety researchers and teams deploying LLM-based planning systems, this establishes a formal bound on what rollout testing guarantees. Acceptance certifies sample consistency; rare rules at mode boundaries remain invisible.