面向扩散模型模块化定制的正交适配
Orthogonal Adaptation for Modular Customization of Diffusion Models
AI总结：
针对多个独立定制扩散模型难以高效合并的问题，提出 Orthogonal Adaptation，通过约束残差权重正交实现无额外开销的多概念联合生成，并在身份保持与效率上优于基线。
摘要：
文本到图像模型的定制技术为一系列此前无法实现的应用铺平了道路，使用户能够在多样的情境和风格中生成特定概念。现有方法虽然能够对单个概念或有限的、预定义的一组概念实现高保真定制，却缺乏可扩展性，无法让单一模型无缝呈现无数概念。本文提出一个名为“模块化定制”的新问题，目标是高效合并针对各个概念独立微调得到的定制模型，使合并后的模型能够在一张图像中联合合成多个概念，同时不损失保真度，也不产生任何额外计算开销。为解决该问题，我们提出 Orthogonal Adaptation（正交适配）方法，旨在促使在微调阶段彼此无法访问信息的定制模型拥有正交的残差权重，从而保证推理时各定制模型可以直接相加且干扰最小。该方法简单通用，适用于模型架构中几乎所有可优化权重。大量定量与定性评估表明，我们的方法在效率和身份保持方面始终优于相关基线，标志着向扩散模型可扩展定制迈出的重要一步。
英文摘要：
Customization techniques for text-to-image models have paved the way for a wide range of previously unattainable applications, enabling the generation of specific concepts across diverse contexts and styles. While existing methods facilitate high-fidelity customization for individual concepts or a limited, pre-defined set of them, they fall short of achieving scalability, where a single model can seamlessly render countless concepts. In this paper, we address a new problem called Modular Customization, with the goal of efficiently merging customized models that were fine-tuned independently for individual concepts. This allows the merged model to jointly synthesize concepts in one image without compromising fidelity or incurring any additional computational costs. To address this problem, we introduce Orthogonal Adaptation, a method designed to encourage the customized models, which do not have access to each other during fine-tuning, to have orthogonal residual weights. This ensures that during inference time, the customized models can be summed with minimal interference. Our proposed method is both simple and versatile, applicable to nearly all optimizable weights in the model architecture. Through an extensive set of quantitative and qualitative evaluations, our method consistently outperforms relevant baselines in terms of efficiency and identity preservation, demonstrating a significant leap toward scalable customization of diffusion models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
