扩散模型中定制化概念的无混淆生成
Non-confusing Generation of Customized Concepts in Diffusion Models
AI总结：
针对文本引导扩散模型定制化概念生成中的视觉混淆问题，提出对比图像-语言微调方法CLIF，通过微调CLIP获取无混淆文本嵌入，有效避免多定制概念生成混淆。
摘要：
我们针对文本引导扩散模型（TGDMs）在组合概念生成中普遍存在的概念间视觉混淆问题展开研究，该问题在定制化概念生成场景下因用户提供的概念视觉样例稀缺而更为突出。通过回顾推动TGDMs成功的两大核心阶段——1）用于编码视觉语义的文本编码器的对比图像-语言预训练（CLIP），2）将文本嵌入解码为像素的TGDM训练，我们指出现有定制化生成方法仅聚焦于第二阶段的微调，却忽略了第一阶段。为此，我们提出一种简单却有效的解决方案CLIF：对比图像-语言微调。具体而言，给定少量定制化概念样本，我们通过将某一概念与其他概念的过分割视觉区域进行对比来微调CLIP，从而获得该概念的无混淆文本嵌入。实验结果证明了CLIF在避免多定制化概念生成混淆方面的有效性。
英文摘要：
We tackle the common challenge of inter-concept visual confusion in compositional concept generation using text-guided diffusion models (TGDMs). It becomes even more pronounced in the generation of customized concepts, due to the scarcity of user-provided concept visual examples. By revisiting the two major stages leading to the success of TGDMs -- 1) contrastive image-language pre-training (CLIP) for text encoder that encodes visual semantics, and 2) training TGDM that decodes the textual embeddings into pixels -- we point that existing customized generation methods only focus on fine-tuning the second stage while overlooking the first one. To this end, we propose a simple yet effective solution called CLIF: contrastive image-language fine-tuning. Specifically, given a few samples of customized concepts, we obtain non-confusing textual embeddings of a concept by fine-tuning CLIP via contrasting a concept and the over-segmented visual regions of other concepts. Experimental results demonstrate the effectiveness of CLIF in preventing the confusion of multi-customized concept generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
