arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.13095cs.CL
收藏

薛定谔的蝙蝠:扩散模型有时会产生叠加的多义词

Schrödinger's Bat: Diffusion Models Sometimes Generate Polysemous Words in Superposition

University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院)

Jennifer C. White, Ryan Cotterell

摘要:

最近的工作已经表明,尽管它们具有令人印象深刻的能力,但是文本到图像扩散模型,例如DALL-E 2(Ramesh等人,2022)可以在提示包含具有多种可能含义的单词时显示奇怪的行为,通常生成包含单词的两种含义的图像(Rassin等人,2022年)。在这项工作中,我们试图提出一个可能的解释这一现象。使用类似的稳定扩散模型(Rombach等人,2022),我们首先表明,当给定一个输入是两个不同的词的编码之和时,模型可以产生一个包含两个概念的图像。然后,我们证明了用于对提示进行编码的CLIP编码器(Radford等人,2021)将多义词编码为意义的叠加,并且使用线性代数技术,我们可以编辑这些表示以影响生成的图像中表示的感官。结合这两个发现,我们认为Rassin等人(2022)描述的同音异义词重复现象是由扩散模型产生的图像引起的,这些图像代表了多义词编码中叠加的两种含义。

英文摘要:

Recent work has shown that despite their impressive capabilities, text-to-image diffusion models such as DALL-E 2 (Ramesh et al., 2022) can display strange behaviours when a prompt contains a word with multiple possible meanings, often generating images containing both senses of the word (Rassin et al., 2022). In this work we seek to put forward a possible explanation of this phenomenon. Using the similar Stable Diffusion model (Rombach et al., 2022), we first show that when given an input that is the sum of encodings of two distinct words, the model can produce an image containing both concepts represented in the sum. We then demonstrate that the CLIP encoder used to encode prompts (Radford et al., 2021) encodes polysemous words as a superposition of meanings, and that using linear algebraic techniques we can edit these representations to influence the senses represented in the generated images. Combining these two findings, we suggest that the homonym duplication phenomenon described by Rassin et al. (2022) is caused by diffusion models producing images representing both of the meanings that are present in superposition in the encoding of a polysemous word.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑