arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.12416cs.CVcs.LG
收藏

Prompting Hard or Hardly Prompting: 面向文本生成图像扩散模型的提示词反演

Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models

University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(Vector人工智能研究所)

Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi, Leonid Sigal

AI总结:

针对文本生成图像扩散模型提示词工程繁琐、现有嵌入反演结果缺乏可解释性的问题,提出基于延迟投影和扩散时间步语义特性的提示词反演方法,可直接生成可解释的语义提示词并应用于多种图像编辑任务。

摘要:

提供给文本生成图像扩散模型的提示词质量决定了生成内容与用户意图的契合度,通常需要进行“提示词工程”。为了在无需提示词工程的情况下从目标图像中提取视觉概念,现有方法主要依赖嵌入反演,即先优化嵌入再将其映射为伪标记(pseudo-token)。然而,这类高维向量表示的处理颇具挑战:它们缺乏语义性与可解释性,使用时仅能进行简单的向量运算。\n为此,本研究聚焦于对扩散模型进行反演,直接获取可解释的语言提示词。这一过程的难点在于,对应的优化问题本质上是离散的,且提示词空间呈指数级庞大,导致随机梯度下降等标准优化技术难以适用。为解决该问题,我们采用延迟投影方案,优化得到能代表模型词汇空间的提示词。此外,我们利用了扩散过程不同时间步对应图像不同细节层次的研究发现:前向扩散过程中较晚的含噪时间步对应语义信息,因此在该范围内进行提示词反演可得到能代表图像语义的标记。\n实验表明,我们的方法能为目标图像识别出具备语义可解释性且有意义的提示词,这些提示词可用于合成内容相似的多样化图像。我们还进一步展示了优化后的提示词在进化式图像生成与概念去除任务中的应用。

英文摘要:

The quality of the prompts provided to text-to-image diffusion models determines how faithful the generated content is to the user's intent, often requiring `prompt engineering'. To harness visual concepts from target images without prompt engineering, current approaches largely rely on embedding inversion by optimizing and then mapping them to pseudo-tokens. However, working with such high-dimensional vector representations is challenging because they lack semantics and interpretability, and only allow simple vector operations when using them. Instead, this work focuses on inverting the diffusion model to obtain interpretable language prompts directly. The challenge of doing this lies in the fact that the resulting optimization problem is fundamentally discrete and the space of prompts is exponentially large; this makes using standard optimization techniques, such as stochastic gradient descent, difficult. To this end, we utilize a delayed projection scheme to optimize for prompts representative of the vocabulary space in the model. Further, we leverage the findings that different timesteps of the diffusion process cater to different levels of detail in an image. The later, noisy, timesteps of the forward diffusion process correspond to the semantic information, and therefore, prompt inversion in this range provides tokens representative of the image semantics. We show that our approach can identify semantically interpretable and meaningful prompts for a target image which can be used to synthesize diverse images with similar content. We further illustrate the application of the optimized prompts in evolutionary image generation and concept removal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑