arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.01327cs.CV
收藏

SPDiffusion:用于多概念文本到图像生成的语义保护扩散模型

SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generation

University of Chinese Academy of Sciences(中国科学院大学)

Yang Zhang, Rui Zhang, Xuecheng Nie, Haochen Li, Jikun Chen, Yifan Hao, Xin Zhang, Luoqi Liu, Ling Li

AI总结:

针对多概念文本到图像生成中的语义纠缠问题,提出SPDiffusion模型,通过SP-Extraction和SP-Attn机制解决概念纠缠与属性绑定问题,在现有基准上取得最先进结果。

摘要:

最近的文本到图像模型在生成高质量图像方面取得了令人瞩目的成果。然而,在处理多概念生成(即创建包含多个角色或对象的图像)时,现有方法经常遭遇语义纠缠问题,包括概念纠缠和不当属性绑定,导致显著的文本与图像不一致。我们发现,当潜在特征的某些区域关注了错误的概念和属性标记时,就会产生语义纠缠。在本研究中,我们提出了语义保护扩散模型(SPDiffusion),仅需输入文本提示即可解决概念纠缠和不当属性绑定问题。SPDiffusion框架引入了一种新颖的概念区域提取方法SP-Extraction,以解决交叉注意力中的区域纠缠,并提出了SP-Attn,保护概念区域免受无关属性和概念的影响。为了评估我们的方法,我们在现有基准上进行了测试,SPDiffusion取得了最先进的结果,证明了其有效性。

英文摘要:

Recent text-to-image models have achieved impressive results in generating high-quality images. However, when tasked with multi-concept generation creating images that contain multiple characters or objects, existing methods often suffer from semantic entanglement, including concept entanglement and improper attribute binding, leading to significant text-image inconsistency. We identify that semantic entanglement arises when certain regions of the latent features attend to incorrect concept and attribute tokens. In this work, we propose the Semantic Protection Diffusion Model (SPDiffusion) to address both concept entanglement and improper attribute binding using only a text prompt as input. The SPDiffusion framework introduces a novel concept region extraction method SP-Extraction to resolve region entanglement in cross-attention, along with SP-Attn, which protects concept regions from the influence of irrelevant attributes and concepts. To evaluate our method, we test it on existing benchmarks, where SPDiffusion achieves state-of-the-art results, demonstrating its effectiveness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑