SeqLoRA: 为持续多概念生成的双水平正则化适应
SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation
AI总结：
本文提出SeqLoRA，一种双水平优化框架，通过联合优化LoRA因素来解决文本到图像扩散模型中多自定义概念组合时的表示干扰问题，提高了身份保持性和可扩展性。
摘要：
参数高效微调能够快速个性化文本到图像扩散模型，但组合多个自定义概念仍然具有挑战性，因为存在表示干扰。现有的模块化方法要么依赖于昂贵的后置融合，要么冻结适应子空间，这限制了表达能力和概念保真度。为了解决这一权衡，我们提出了顺序正则化的LoRA（SeqLoRA），一种联合优化LoRA因素的持续学习框架。理论上，我们为我们的算法建立了强收敛保证，并将残差层激活建模为矩阵子高斯过程，以推导出灾难性遗忘的高概率界。我们进一步证明，从数据中学习LoRA基底比冻结基底方法更有效地最小化残差干扰能量。在多概念图像生成实验中，SeqLoRA在多达101个概念上提高了身份保持性和可扩展性，同时避免了昂贵的融合并减少了组合生成中的属性干扰。
英文摘要：
Parameter-efficient fine-tuning enables fast personalization of text-to-image diffusion models to user-provided concepts (objects, people, or styles), but composing multiple such concepts remains challenging due to representation interference. Existing modular methods, usually built on low-rank adaptation (LoRA), either rely on expensive post-hoc fusion or freeze the LoRA adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization while keeping each new basis orthogonal to all previously learned ones. Theoretically, we establish monotone descent and convergence to a critical point of the constrained problem, and model the residual layer activations as a matrix sub-Gaussian process to derive a high-probability bound on catastrophic forgetting in multi-layer nonlinear networks. This bound depends on the basis only through a residual interference energy, and within the feasible subspace we prove that a data-adapted basis minimizes it, whereas a random frozen basis is suboptimal in expectation. Experiments on Stable Diffusion with up to 101 concepts show that SeqLoRA improves identity preservation over fusion-based methods, attains the lowest cross-concept leakage in multi-concept compositions, requires no fusion step, and scales to concept counts at which fusion runs out of memory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
