LoRAForge: Co-Designing Post-Hoc Compression and Execution for Efficient Multi-LoRA Serving
| Source: arXiv AI
Tags: LoRAForge, LoRA, multi-LoRA serving, FlashTSQR, Qwen, Llama
LoRAForge compresses trained LoRA adapters and pairs them with rank-specialized kernels so smaller adapters actually serve faster, keeping about 7.7% of parameters with at most a 1.2-point score drop.
Details
Serving many LoRA adapters on one base model adds storage, loading and execution overhead, and post-hoc compression alone does not speed things up because padding and generic kernels hide the reduced rank. LoRAForge links batched factor construction to rank-specialized execution through an explicit retained-rank representation, with uniform (LoRAForge-U) and spectral-energy (LoRAForge-S) allocation policies. Its FlashTSQR step speeds factor construction up to 2.28x over matched library QR. In a Qwen control the runtime added 1.17 to 1.22x throughput. Across 85 Qwen and Llama adapters, the spectral policy kept about 7.7% of adapter parameters with fleet-average drops of at most 1.2 points out of 100, and the 1/12 exports gained token throughput on both AMD and NVIDIA GPUs.