Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs

Details

MirroS introduced Code-as-World, a system that represents physical scenes not as pixels or latents but as executable code—a scene.json MuJoCo file encoding composition (geometry, mass, friction), evolution (forces, contacts), and appearance (camera, lighting). The core claim: pixels are evidence of a physical scene, not its ontology, so video models that predict frames sidestep actual physics. To recover a physics program from real footage, the system runs an abductive search loop up to five rounds: propose a program, execute it in MuJoCo, render frames, and compare against the source video using RGB, depth, mask, and trajectory discrepancies. SAM 3 supplies instance masks, VGGT-Omega estimates depth and camera geometry, and SAM 3D generates per-object meshes. Structured feedback from each round guides iterative program refinement. The verified EWR programs become training data with exact physical labels—something raw video inherently lacks. Code-as-World-VL-9B, fine-tuned from Qwen3.5-9B on this synthetic data, scores 55.4 MRA on QuantiPhy-validation, above Gemini-3.1 Flash at 54.8 and roughly 15 points above the strongest open-weight baseline. Both models (4B and 9B) ship as Apache 2.0 BF16 safetensors, deployable via vLLM behind an OpenAI-compatible /v1 endpoint. The approach opens a path toward simulation-grounded robotics training without expensive manual annotation.