Locking Pretrained Weights via Deep Low-Rank Residual Distillation

August 6, 2026 | Source: Apple ML Research

Tags: Apple, model security, fine-tuning defense, open-weight models, DLR-Lock, weight protection

Apple researchers propose DLR-Lock, which replaces model MLPs with deep low-rank residual networks to make fine-tuning computationally prohibitive — a defense against unauthorized adaptation of open-weight models that holds against attackers with full knowledge of the strategy.

Details

DLR-Lock (Deep Low-Rank Lock) exploits the asymmetry between inference and training in automatic differentiation. Each MLP in a pretrained model is replaced with a Deep Low-Rank Residual Network (DLR-Net) of similar parameter count. DLR-Nets are efficiently trained via module-wise distillation to preserve original model capabilities — but their architecture forces activation memory to grow linearly with depth during backpropagation. This memory overhead makes fine-tuning disproportionately expensive relative to inference. The defense is designed to hold against adaptive attackers who know the full model architecture and defense strategy — a meaningful security bar, since simple obfuscation fails trivially when weights are public. The practical use case: model publishers who want to share weights for inference but prevent downstream fine-tuning for unauthorized purposes. This tension is growing as capable open-weight models proliferate and commercial licenses increasingly restrict fine-tuning. Experiments on LLMs validate that capabilities are preserved at inference time while fine-tuning is practically hampered.