Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
| Source: Apple ML Research
Tags: Apple, Siri, text-to-speech, Diffusion Transformer, on-device AI, audio synthesis, AFM, RVQ
Apple's production TTS system for Siri Expressive Voices achieves 16x real-time audio synthesis on-device with just 21MB peak runtime memory, using a Diffusion Transformer architecture that decouples temporal and depth processing for constant-memory inference.
Details
Apple ML Research has published the architecture behind Siri Expressive Voices — the voice synthesis system now shipping in production on Apple devices with customizable Pace and Expressivity sliders. The system runs entirely on the Apple Matrix Coprocessor (AMX), powered by AFM 3 Core Advanced at a 1-billion-parameter activation size. The key architectural innovation is a three-component detokenizer — streaming encoder, temporal decoder, and depth decoder — that converts the foundation model's semantic audio tokens into waveform audio. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning replaces the dedicated per-level decoders used in prior multi-decoder architectures. Causal sliding window attention with fixed-window key-value caching keeps memory complexity constant regardless of sequence length, enabling synthesis of 20–320 second audio clips with only ~21MB peak runtime memory and 329MB of on-device assets. Benchmarks show the system runs at ~10ms per generation step (16x faster than real time) while improving Mean Opinion Score by +0.28 overall (from 3.87 to 4.15) and +0.42 on conversational speech specifically. For practitioners building on-device audio generation, the paper demonstrates that constant-memory DiT architectures can match or exceed GAN and conventional transformer quality while remaining deployable within tight mobile compute constraints.