SaMA: Morpho Adaptation via Asymmetric Expansion of Kronecker Product

| Source: arXiv AI

Tags: SaMA, LoRA, PEFT, Kronecker Product, NeurIPS 2026

SaMA, a NeurIPS 2026 parameter-efficient fine-tuning method built on a generalized Kronecker product, scales effective rank quadratically with parameter budget and beats strong PEFT baselines on reasoning benchmarks with fewer parameters.

Details

LoRA dominates parameter-efficient fine-tuning, but the authors point to two limits: its invariance manifold creates flat directions that slow adaptive optimizers, and its effective rank grows only linearly with the parameter budget. Scalable Morpho Adaptation (SaMA) relaxes the shared-block constraint in the perfect-shuffle factorization of the Kronecker product. This lets it span low to full rank, with practical rank scaling quadratically with the budget. Its invariance manifold collapses to a diagonal group, which the authors argue makes it friendlier to optimizers. SaMA outperformed strong PEFT baselines on commonsense and arithmetic reasoning benchmarks while using fewer parameters. The paper is accepted at NeurIPS 2026.