LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

August 18, 2026 | Source: arXiv AI

Tags: PEFT, fine-tuning, LoRA, LLaMA, Tucker-decomposition, parameter-efficient, LLM

LORA-CRAFT (CRAFT) achieves LoRA-level or better accuracy on LLaMA2-7B and LLaMA3-8B using hundreds of times fewer trainable parameters, by applying Tucker tensor decomposition to pre-trained attention weights across layers and training only small adaptation matrices on frozen factors.

Details

Parameter-efficient fine-tuning (PEFT) is critical for adapting large language models without full retraining. LoRA dominates the field by adding trainable low-rank matrices to weight updates. CRAFT takes a fundamentally different direction: instead of decomposing gradient updates (as LoTR and SuperLoRA do), it applies Tucker decomposition directly to the pre-trained attention weights organized as 3D tensors stacked across transformer layers. The decomposition is computed via Higher-Order SVD (HOSVD) on the stacked pre-trained weights. All resulting Tucker factors are then frozen. The trainable portion is only small square adaptation matrices applied to each factor — a dramatically smaller parameter budget than LoRA. On the GLUE benchmark with RoBERTa-base and RoBERTa-large, and on commonsense reasoning with LLaMA2-7B and LLaMA3-8B, CRAFT competes with or exceeds LoRA at parameter counts hundreds of times smaller. The efficiency advantage appears to grow with model scale — suggesting the Tucker factors better capture the richer cross-layer structure of larger pre-trained models. For practitioners, this matters most in fine-tuning with very limited compute budgets, or multi-task settings where per-task parameter count must be minimal. The approach requires pre-computing the Tucker decomposition upfront, but that cost is amortized across fine-tuning runs.