arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.01154cs.CVcs.AI
收藏

揭示文本到图像扩散模型中的文本嵌入

Uncovering the Text Embedding in Text-to-Image Diffusion Models

University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团)

Hu Yu, Hao Luo, Fan Wang, Feng Zhao

AI总结:

本文深入分析文本到图像扩散模型中的文本嵌入空间,揭示词嵌入重要性、上下文相关性及通过SVD发现的语义潜力,为免学习图像编辑和语义发现提供指导。

摘要:

输入文本与生成图像之间的对应关系呈现出不透明性,其中微小的文本修改可能导致生成图像产生显著偏差。然而,文本嵌入作为文本与图像之间的关键中介,仍然相对未被充分探索。在本文中,我们通过深入研究文本嵌入空间来填补这一研究空白,在免学习框架下释放其用于可控图像编辑和可解释语义方向属性的能力。具体而言,我们确定了关于每个词嵌入的重要性及其在文本嵌入中的上下文相关性的两个关键见解,为免学习图像编辑提供了指导性原则。此外,我们发现文本嵌入本质上具有多样的语义潜力,并通过奇异值分解(SVD)的视角进一步揭示了这一特性。这些被揭示的特性为图像编辑和语义发现提供了实用价值。更重要的是,我们期望对文本嵌入的深入分析和发现能够增强对文本到图像扩散模型的理解。

英文摘要:

The correspondence between input text and the generated image exhibits opacity, wherein minor textual modifications can induce substantial deviations in the generated image. While, text embedding, as the pivotal intermediary between text and images, remains relatively underexplored. In this paper, we address this research gap by delving into the text embedding space, unleashing its capacity for controllable image editing and explicable semantic direction attributes within a learning-free framework. Specifically, we identify two critical insights regarding the importance of per-word embedding and their contextual correlations within text embedding, providing instructive principles for learning-free image editing. Additionally, we find that text embedding inherently possesses diverse semantic potentials, and further reveal this property through the lens of singular value decomposition (SVD). These uncovered properties offer practical utility for image editing and semantic discovery. More importantly, we expect the in-depth analyses and findings of the text embedding can enhance the understanding of text-to-image diffusion models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑