Taming Outlier Tokens in Diffusion Transformers
| Source: Apple ML Research
Tags: Apple, Diffusion Transformers, image generation, outlier tokens, Vision Transformers, DiT
Apple researchers find that Diffusion Transformers develop high-norm outlier tokens in intermediate layers that attract excess attention but carry degraded local semantics — and show that simply masking them fails; their Dual-Stage Registers (DSR) fix both encoder and denoiser stages to improve image generation quality.
Details
Prior work established that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited useful information. This Apple/Rice University paper shows the same phenomenon appears in Representation Autoencoder + Diffusion Transformer (RAE-DiT) pipelines: pretrained ViT encoders produce outlier representations, and the DiT denoiser develops its own internal outlier tokens, especially in intermediate layers. The key finding is that masking high-norm tokens does not improve generation — the problem is not simply a few extreme values but rather corrupted local patch semantics. This distinction matters for practitioners: approaches that detect and suppress outliers by magnitude alone will not address the root cause. The proposed fix, Dual-Stage Registers (DSR), intervenes at both components: trained registers when available, recursive test-time registers otherwise, and diffusion registers for the denoiser. Tested on ImageNet and large-scale text-to-image generation, DSR consistently reduces outlier artifacts and improves generation quality. The authors frame outlier-token control as an important ingredient in building stronger DiTs — relevant for anyone training or fine-tuning image generation models.