ClassDiffusion：利用显式类别指导实现更对齐的个性化微调
ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance
AI总结：
本文针对文本到图像个性化微调中的概念语义漂移与组合能力下降问题，提出 ClassDiffusion，通过语义保持损失显式约束概念空间，提升图像及个性化视频生成的组合表现。
摘要：
近期的文本到图像定制工作已证明，通过在少量示例上微调扩散模型，可以成功生成给定概念的图像。然而，基于微调的方法本质上倾向于对概念过拟合，导致无法在多种条件下创建该概念（例如，在生成“一只戴着耳机的狗”时耳机缺失）。有趣的是，我们注意到微调前的基础模型具备将基础概念与其他元素组合的能力（例如，“一只戴着耳机的狗”），这意味着组合能力仅在个性化微调后消失。我们观察到定制概念在微调后发生语义偏移，表明个性化概念与原始概念不再对齐，并进一步通过理论分析表明，这种语义偏移会增加采样联合条件概率分布的难度，从而导致组合能力丧失。受这一发现启发，我们提出 ClassDiffusion，该技术利用语义保持损失，在学习新概念时显式约束概念空间。尽管简单，但该方法有效防止了目标概念微调过程中的语义漂移。大量定性和定量实验表明，使用语义保持损失可有效提升微调模型的组合能力。最后，我们还将 ClassDiffusion 扩展到个性化视频生成，证明了其灵活性。
英文摘要：
Recent text-to-image customization works have proven successful in generating images of given concepts by fine-tuning diffusion models on a few examples. However, tuning-based methods inherently tend to overfit the concepts, resulting in failure to create the concept under multiple conditions (*e.g.*, headphone is missing when generating "a `dog wearing a headphone"). Interestingly, we notice that the base model before fine-tuning exhibits the capability to compose the base concept with other elements (*e.g.*, "a dog wearing a headphone"), implying that the compositional ability only disappears after personalization tuning. We observe a semantic shift in the customized concept after fine-tuning, indicating that the personalized concept is not aligned with the original concept, and further show through theoretical analyses that this semantic shift leads to increased difficulty in sampling the joint conditional probability distribution, resulting in the loss of the compositional ability. Inspired by this finding, we present **ClassDiffusion**, a technique that leverages a **semantic preservation loss** to explicitly regulate the concept space when learning a new concept. Although simple, this approach effectively prevents semantic drift during the fine-tuning process of the target concepts. Extensive qualitative and quantitative experiments demonstrate that the use of semantic preservation loss effectively improves the compositional abilities of fine-tuning models. Lastly, we also extend our ClassDiffusion to personalized video generation, demonstrating its flexibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
