Bringing Nunchaku 4-bit Diffusion Inference to Diffusers
| Source: Hugging Face Blog
Tags: Diffusers, Nunchaku, SVDQuant, quantization, diffusion-models, NVFP4, Hugging Face
Hugging Face integrates the Nunchaku inference engine into Diffusers, enabling W4A4 (4-bit weights and activations) quantization for large diffusion transformers via a standard from_pretrained() call — reducing the 20-30 GB VRAM requirement while also accelerating the denoising loop, unlike weight-only backends that only trade precision for memory.
Details
Large text-to-image diffusion models typically require 20-30 GB of VRAM in BF16 precision, putting them out of reach for most consumer GPUs. Diffusers already integrates several quantization backends — bitsandbytes, GGUF, torchao, Quanto — but all of these are weight-only approaches: they store weights at lower precision and dequantize back to full precision at compute time. This cuts memory usage but often adds latency rather than reducing it. SVDQuant, the method behind the Nunchaku inference engine, takes a different approach. It runs the main transformer layers with 4-bit weights AND activations (W4A4), which both reduces memory and speeds up the denoising loop. Previously, using Nunchaku checkpoints required a completely separate inference library outside the Diffusers ecosystem. As of today, Nunchaku checkpoints load natively through Diffusers' standard from_pretrained() API using the new kernels package. The NVFP4 kernels are downloaded automatically from the Hugging Face Hub on first use — no local CUDA compilation required. A companion toolkit called diffuse-compressor lets developers quantize their own architectures and publish them as regular Diffusers repositories. Pre-quantized checkpoints pairing a Nunchaku NVFP4 transformer with a bitsandbytes NF4 text encoder are already available on the Hub. The integration was authored by Pham Hong Vinh (rootonchair) and Sayak Paul from Hugging Face.