Mix-of-Show：用于扩散模型多概念定制的去中心化低秩适应
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
AI总结：
Mix-of-Show提出去中心化低秩适应框架，通过ED-LoRA和梯度融合解决多概念定制中的概念冲突与身份丢失，实现高保真组合。
摘要：
公开的大规模文本到图像扩散模型，如Stable Diffusion，已引起社区的广泛关注。这些模型可以通过低秩适应（LoRA）轻松定制新概念。然而，利用多个概念LoRA来联合支持多个定制概念带来了挑战。我们将此场景称为去中心化多概念定制，它涉及单客户端概念调优和中心节点概念融合。在本文中，我们提出了一个名为Mix-of-Show的新框架，以解决去中心化多概念定制的挑战，包括由现有单客户端LoRA调优导致的概念冲突和模型融合过程中的身份丢失。Mix-of-Show采用嵌入分解的LoRA（ED-LoRA）进行单客户端调优，并在中心节点使用梯度融合，以保留单概念的在域本质并支持理论上无限的概念融合。此外，我们引入了区域可控采样，该采样扩展了空间可控采样（如ControlNet和T2I-Adaptor），以解决多概念采样中的属性绑定和对象缺失问题。大量实验表明，Mix-of-Show能够以高保真度组合多个定制概念，包括角色、对象和场景。
英文摘要：
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community. These models can be easily customized for new concepts using low-rank adaptations (LoRAs). However, the utilization of multiple concept LoRAs to jointly support multiple customized concepts presents a challenge. We refer to this scenario as decentralized multi-concept customization, which involves single-client concept tuning and center-node concept fusion. In this paper, we propose a new framework called Mix-of-Show that addresses the challenges of decentralized multi-concept customization, including concept conflicts resulting from existing single-client LoRA tuning and identity loss during model fusion. Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, we introduce regionally controllable sampling, which extends spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address attribute binding and missing object problems in multi-concept sampling. Extensive experiments demonstrate that Mix-of-Show is capable of composing multiple customized concepts with high fidelity, including characters, objects, and scenes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
