DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization

Details

DiFlowDubber is a new approach to automated video dubbing that addresses limitations in existing methods, which often rely on limited datasets or struggle with expressive speech generation. This system utilizes a two-stage training framework to transfer knowledge from pre-trained text-to-speech (TTS) models to video dubbing. Key components include the FaPro module, which captures prosody and stylistic cues from facial expressions, and a Synchronizer module that improves alignment between text, video, and speech. Experimental results indicate that DiFlowDubber outperforms previous dubbing methods on benchmark datasets, demonstrating its effectiveness in generating temporally synchronized and expressive speech.