FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

Details

High-resolution image-to-video generation fails in predictable ways: generating at a model's native (lower) resolution loses fine detail, while tiled high-resolution denoising preserves local detail but breaks global consistency — scenes fragment, characters drift between tiles. FrescoDiffusion solves this with a two-pass approach. First, generate a low-resolution video at the model's native resolution. Upsample its latent trajectory to create a global prior capturing long-range temporal and spatial structure. Then, for 4K generation, run per-tile noise predictions but fuse them with this prior at every diffusion timestep through a weighted least-squares objective in model-output space. The fusion combines a standard tile-merging criterion with a regularization term from the prior, yielding a closed-form update that requires no additional training. An additional spatial regularization variable lets users specify per-region motion control — choosing which parts of a complex image should animate and which should remain static. The paper targets the 'fresco animation' problem — bringing large complex artworks with many distinct sub-scenes to life while maintaining spatial coherence over time. It evaluates on VBench-I2V and a new fresco dataset, outperforming tiled baselines on global consistency. Training-free and compatible with any existing I2V diffusion model.