arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.09966cs.CV
收藏

文本定位:分解多概念图像以用于主体驱动的文本到图像生成

Textual Localization: Decomposing Multi-concept Images for Subject-Driven Text-to-Image Generation

Durham University(杜伦大学)

Junjie Shentu, Matthew Watson, Noura Al Moubayed

AI总结:

针对主体驱动模型在多概念输入图像中难以指定目标概念的问题,提出一种文本定位模型,通过新颖的交叉注意力引导在微调时分解多概念,显著提升了图像保真度与图文对齐效果。

摘要:

主体驱动的文本到图像扩散模型使用户能够利用少量样本图像,将模型适配到预训练数据集中缺失的新概念。然而,现有的主体驱动模型主要依赖单概念输入图像,在处理多概念输入图像时,难以指定目标概念。为此,我们引入了一种文本定位的文本到图像模型(Textual Localization)来处理多概念输入图像。在微调期间,我们的方法引入了一种新颖的交叉注意力引导来分解多个概念,在目标概念的视觉表示与文本提示中的标识符标记之间建立明确的联系。实验结果表明,在多概念输入图像的图像保真度和图文对齐方面,我们的方法优于或与基线模型表现相当。与Custom Diffusion相比,我们采用硬引导的方法在单概念和多概念生成中,CLIP-I得分分别提高了7.04%和8.13%,CLIP-T得分分别提高了2.22%和5.85%。值得注意的是,我们的方法生成的交叉注意力图与生成图像中的目标概念一致,这是现有模型所不具备的能力。

英文摘要:

Subject-driven text-to-image diffusion models empower users to tailor the model to new concepts absent in the pre-training dataset using a few sample images. However, prevalent subject-driven models primarily rely on single-concept input images, facing challenges in specifying the target concept when dealing with multi-concept input images. To this end, we introduce a textual localized text-to-image model (Texual Localization) to handle multi-concept input images. During fine-tuning, our method incorporates a novel cross-attention guidance to decompose multiple concepts, establishing distinct connections between the visual representation of the target concept and the identifier token in the text prompt. Experimental results reveal that our method outperforms or performs comparably to the baseline models in terms of image fidelity and image-text alignment on multi-concept input images. In comparison to Custom Diffusion, our method with hard guidance achieves CLIP-I scores that are 7.04%, 8.13% higher and CLIP-T scores that are 2.22%, 5.85% higher in single-concept and multi-concept generation, respectively. Notably, our method generates cross-attention maps consistent with the target concept in the generated images, a capability absent in existing models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑