MLLM-Guided Semantic Correction for Text-to-Video Generation

Details

Text-to-video diffusion models frequently produce semantic errors — missing objects, incorrect attributes, mismatched actions — that existing correction methods address either before or after sampling, not during. This paper introduces a training-free, interpretable correction framework that integrates multimodal LLM (MLLM) feedback directly into the diffusion sampling process. Two modules do the work: a Semantic Assessment Supervisor generates intermediate preview frames at selected timesteps, runs semantic evaluation and deviation diagnostics using an MLLM, and produces corrective signals. A Semantic Modification Assistant then injects these signals into the diffusion trajectory via latent space intervention, steering generation back toward the intended semantics without restarting the process. The system improves semantic alignment, visual fidelity, and temporal consistency across multiple benchmarks without modifying model parameters. The training-free aspect is practically significant — it can be applied to any existing text-to-video diffusion model as a plugin. The MLLM also provides interpretable diagnostic output alongside the correction, useful for debugging generation failures. Specific quantitative improvements relative to baselines are not reported in the abstract. Practitioners considering adoption should check the paper's benchmark tables for magnitude of gains before deploying in production.