OmniOPD: Stable On-Policy Distillation for Joint Audio-Video Generation
| Source: arXiv AI
Tags: OmniOPD, on-policy distillation, audio-video generation, JavisBench, VBench
OmniOPD stabilizes on-policy distillation for joint audio-video generation by giving the audio teacher cleaner visual context and more audio supervision, improving video quality while keeping audio quality.
Details
RL improves joint audio-video generation, but separate audio and video rewards interfere. On-policy distillation from specialized teachers helps, yet audio gets sparse supervision because it has far fewer tokens than video. OmniOPD uses asymmetric noise scheduling to give the audio teacher privileged, cleaner video context, grouped multi-noise sampling to widen audio supervision, and timestep-wise importance weighting based on teacher-student divergence to down-weight unreliable signals. On JavisBench and VBench, the authors report preserved audio quality and clearly better video generation.