选择性信息描述可以减少文本到图像个性化中不需要的嵌入纠缠
Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization
AI总结:
针对文本到图像个性化中参考图像偏差导致主体嵌入纠缠的问题,提出利用多模态GPT-4生成的选择性信息描述(SID)策略,有效减少不需要的嵌入纠缠并提升图像与提示的对齐度。
摘要:
在文本到图像个性化中,一个及时且关键的挑战是生成的图像容易过拟合到参考图像中存在的偏差。我们首先对这些偏差进行全面的分类,将其分为背景、附近物体、关联物体、风格重语境化中的材质以及姿态偏差。由于这些偏差与主体嵌入发生纠缠,它们在生成的图像中显现出来。这种不需要的嵌入纠缠不仅导致参考图像的偏差反映到生成的图像中,还显著降低了生成图像与给定生成提示的对齐度。为了应对这一挑战,我们提出了SID(Selectively Informative Description,选择性信息描述),这是一种文本描述策略,与仅描述主体类别标识的主流方法不同。SID利用多模态GPT-4生成,可以无缝集成到基于优化的模型中。我们展示了全面的实验结果,并对交叉注意力图、主体对齐、非主体解耦和文本对齐进行了分析。
英文摘要:
In text-to-image personalization, a timely and crucial challenge is the tendency of generated images overfitting to the biases present in the reference images. We initiate our study with a comprehensive categorization of the biases into background, nearby-object, tied-object, substance (in style re-contextualization), and pose biases. These biases manifest in the generated images due to their entanglement into the subject embedding. This undesired embedding entanglement not only results in the reflection of biases from the reference images into the generated images but also notably diminishes the alignment of the generated images with the given generation prompt. To address this challenge, we propose SID~(Selectively Informative Description), a text description strategy that deviates from the prevalent approach of only characterizing the subject's class identification. SID is generated utilizing multimodal GPT-4 and can be seamlessly integrated into optimization-based models. We present comprehensive experimental results along with analyses of cross-attention maps, subject-alignment, non-subject-disentanglement, and text-alignment.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。