LoRAForge: Co-Designing Post-Hoc Compression and Execution for Efficient Multi-LoRA Serving

| Source: arXiv AI

Tags: LoRAForge, LoRA, multi-LoRA serving, FlashTSQR, Qwen, Llama

LoRAForge compresses trained LoRA adapters and pairs them with rank-specialized kernels so smaller adapters actually serve faster, keeping about 7.7% of parameters with at most a 1.2-point score drop.

Details

Serving many LoRA adapters on one base model adds storage, loading and execution overhead, and post-hoc compression alone does not speed things up because padding and generic kernels hide the reduced rank. LoRAForge links batched factor construction to rank-specialized execution through an explicit retained-rank representation, with uniform (LoRAForge-U) and spectral-energy (LoRAForge-S) allocation policies. Its FlashTSQR step speeds factor construction up to 2.28x over matched library QR. In a Qwen control the runtime added 1.17 to 1.22x throughput. Across 85 Qwen and Llama adapters, the spectral policy kept about 7.7% of adapter parameters with fleet-average drops of at most 1.2 points out of 100, and the 1/12 exports gained token throughput on both AMD and NVIDIA GPUs.