SFA: A Statistical Physics-Inspired Diffusion Transformer for Efficient and High-Quality Image Generation
| Source: arXiv AI
Tags: Scale-Free Attention, diffusion transformers, FlashAttention, Triton, Stable Diffusion 3.5, PixelFlow
Peking University researchers propose Scale-Free Attention, an O(N log N) attention for diffusion transformers that reports 357x forward speedups over FlashAttention at 262K tokens while slightly improving ImageNet FID.
Details
Scale-Free Attention (SFA) keeps fine local interactions while summarizing distant regions at progressively coarser scales, inspired by scale-free laws in statistical physics. Mean pooling, token-count correction and a shared softmax over the hierarchy give O(N log N) complexity in forward and backward passes. The Triton implementation reports 357.3x forward and 265.6x backward attention speedups over FlashAttention at 262,144 tokens. After 40 epochs of PixelFlow training on ImageNet 256x256, FID-50k improves from 13.25 with full attention to 12.51, and the advantage holds through 80 epochs. Swapping SFA into Stable Diffusion 3.5 Medium reportedly retains the full-attention model's generation quality.