FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training

| Source: arXiv AI

Tags: FP4, Quantization, Model Parallelism, NVIDIA H100, Qwen-7B, FWHT

FP4Quant compresses inter-GPU communication in model-parallel LLM training to 4-bit, reporting up to 1.56x end-to-end throughput over BF16 on H100s with Qwen-7B, with a trade-off in model quality.

Details

Communication overhead is a major bottleneck in distributed model-parallel LLM training, and pushing compression down to FP4 runs into limited numeric range, sub-byte storage overhead and extra quantization compute. FP4Quant combines micro-grouped dual-scale FP4 quantization, warp-cooperative FP4 packing for sub-byte data, and a hybrid two-level FWHT pipeline to reduce quantization error with GPU-aware execution. On NVIDIA H100 GPUs with Qwen-7B, the authors report up to 1.56x end-to-end throughput over BF16, while noting a trade-off between throughput and model quality. The paper was accepted at IFIP NPC 2026.