Marionette: Predicting World States, Rendering Geometry, Painting Appearance

August 17, 2026 | Source: arXiv AI

Tags: world models, game AI, video diffusion, 3D simulation, character animation, computer vision, controllable generation

Marionette separates interactive game world modeling into explicit 3D state prediction, zero-parameter geometric rendering, and video-diffusion appearance synthesis—enabling direct rule injection (terrain collision, character separation) that reduces ground penetration by 66% without retraining any neural component.

Details

Most interactive game world models predict future frames directly in pixel or latent space, forcing the model to implicitly maintain all structural properties (pose, geometry, occlusion). This works short-term but accumulates errors over long horizons: characters drift apart, penetrate the ground, and lose behavioral consistency. Marionette explicitly separates three concerns. A two-stage autoregressive dynamics model predicts a 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. A zero-parameter graphics bridge converts this explicit state to pose-control videos using closed-form geometry and occlusion computation—no neural components, fully deterministic. A video-diffusion observation model synthesizes photorealistic RGB frames from the structured controls. The explicit state representation enables direct rule injection without retraining: adding a terrain collider and a character separation cap reduces ground penetration by 66% and keeps characters engaged. Without rules, characters drift to 21.2m apart; real sessions stay near 5m. Controllability is validated: swapping action streams changes root-aligned joint error by 31% across 48 held-out segments. Appearance quality (FVD of 831 vs. 799 for recorded pose) shows minimal cost from routing appearance through the explicit state. This architecture has direct implications for game AI, simulation, and synthetic training data generation, where controllable long-horizon behavior is critical.