SDO: Subspace Deconflicting Operator for Multi-Adapter Composition
| Source: arXiv AI
Tags: LoRA, diffusion models, multi-adapter, image generation, ACM MM 2026, parameter efficiency
SDO (Subspace Deconflicting Operator) fixes identity mixing and attribute leakage when composing multiple LoRA adapters in diffusion models, plugging into existing inference pipelines without retraining and improving identity fidelity as the number of jointly composed adapters increases.
Details
When multiple LoRA adapters trained on different characters are composed in a shared diffusion backbone, their overlapping parameter subspaces cause identity mixing, cross-character attribute leakage, and unstable scene composition — a common pain point in production multi-character image generation. SDO addresses this by extracting compact subspace signatures from each adapter's low-rank updates, measuring pairwise conflict through output-subspace overlap, and applying a permutation-equivariant transformation that suppresses harmful shared directions while retaining identity-specific characteristics. The modified adapter updates slot directly into existing diffusion inference pipelines with no retraining required. Experiments show consistent gains in identity fidelity and compositional stability, with improvements scaling as the number of simultaneously composed adapters grows — the use case where the problem is most severe. The paper is accepted at ACM MM 2026. Relevant for any creative pipeline using multiple LoRA adapters: character consistency, style mixing, or product visualization at scale.