持续扩散：利用C-LoRA实现文本到图像扩散模型的持续定制
Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
AI总结：
针对文本到图像扩散模型在顺序添加新概念时出现灾难性遗忘的问题，本文提出C-LoRA方法，通过在Stable Diffusion交叉注意力层中引入持续自正则化低秩适应，以极少参数开销实现无数据回放的持续定制，并在图像分类无回放持续学习中取得新最优性能。
摘要：
近期研究展示了仅需提供少量示例图像即可显著定制文本到图像扩散模型的能力。如果尝试以顺序（即持续）方式使用多个细粒度概念来定制此类模型，会发生什么？在我们的工作中，我们表明，当新概念顺序到达时，近期最先进的文本到图像模型定制方法会遭受灾难性遗忘。具体而言，当添加新概念时，生成过去相似概念的高质量图像的能力会下降。为避免这种遗忘，我们提出了一种新方法C-LoRA，该方法由流行的Stable Diffusion模型交叉注意力层中的持续自正则化低秩适应组成。此外，我们使用不包含定制对象词语（例如，对于人脸数据集不使用“person”一词）的定制提示，并将其初始化为完全随机的嵌入。重要的是，我们的方法仅引入极少的额外参数成本，并且不需要存储用户数据进行回放。我们表明，C-LoRA不仅在我们提出的文本到图像持续定制设置（我们称之为Continual Diffusion）中优于多个基线，而且我们在成熟的图像分类无回放持续学习设置中取得了新的最先进性能。C-LoRA在两个独立领域中的高实现性能使其成为广泛应用的引人注目的解决方案，我们相信它具有显著的实际影响潜力。项目页面：https://jamessealesmith.github.io/continual-diffusion/
英文摘要：
Recent works demonstrate a remarkable ability to customize text-to-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.e., continual) manner? In our work, we show that recent state-of-the-art customization of text-to-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, we propose a new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (i.e., "person" for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two separate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact. Project page: https://jamessealesmith.github.io/continual-diffusion/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
