Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

August 18, 2026 | Source: arXiv AI

Tags: voice cloning, text-to-audio-video, generative AI, diffusion models, speaker verification, multimodal

A single zero-initialized linear layer added to a 5B text-to-audio-video model enables state-of-the-art voice cloning across 30 speakers — outperforming five TTS baselines on speaker similarity — and the audio-only path runs approximately 30x faster than full audio-video diffusion.

Details

Text-to-audio-video models generate synchronized video and audio from text descriptions, but have no mechanism for controlling whose voice is used. This paper from Ivan Mikheev and colleagues shows that a single zero-initialized linear layer on top of the audio backbone, fine-tuned on voice cloning data, is sufficient to add speaker control without degrading base model capabilities. The zero-initialization is key: it prevents the new layer from disrupting the pretrained model at the start of fine-tuning, allowing the model to retain its base capabilities while learning the new voice-conditioning task. Reference voice injection uses two complementary paths: diffusion latents are prepended to the audio stream, and a global speaker embedding modulates the target audio token. Evaluated on a benchmark of 674 speaker-text pairs spanning 30 speakers, the 5B model achieves the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming all five TTS baselines. A practical side effect: running only the audio path without the video diffusion loop provides approximately 30x speedup while preserving voice-cloning quality — effectively yielding a competitive TTS model from a T2AV model. The zero-init trick for adding new capabilities is reusable beyond this specific application.