获取你想要的内容,而非你不想要的内容:文本到图像扩散模型的图像内容抑制
Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models
AI总结:
针对文本到图像扩散模型难以抑制提示中未要求内容的问题,提出软加权正则化与推理时文本嵌入优化方法,有效抑制不期望内容并促进期望内容生成,且适用于像素空间与潜在空间模型。
摘要:
近期文本到图像扩散模型的成功很大程度上归因于其能够被复杂的文本提示引导,这使得用户能够精确描述期望内容。然而,这些模型难以有效抑制生成不希望的内容,这些内容在提示中被明确要求从生成图像中省略。在本文中,我们分析了如何操纵文本嵌入并从其中移除不想要的内容。我们提出了两项贡献,分别称为软加权正则化和推理时文本嵌入优化。第一种方法对文本嵌入矩阵进行正则化,有效抑制不希望的内容。第二种方法旨在进一步抑制提示中不希望内容的生成,并鼓励期望内容的生成。我们在大量实验中对我们的方法进行了定量和定性评估,验证了其有效性。此外,我们的方法可推广到像素空间扩散模型(即DeepFloyd-IF)和潜在空间扩散模型(即Stable Diffusion)。
英文摘要:
The success of recent text-to-image diffusion models is largely due to their capacity to be guided by a complex text prompt, which enables users to precisely describe the desired content. However, these models struggle to effectively suppress the generation of undesired content, which is explicitly requested to be omitted from the generated image in the prompt. In this paper, we analyze how to manipulate the text embeddings and remove unwanted content from them. We introduce two contributions, which we refer to as $\textit{soft-weighted regularization}$ and $\textit{inference-time text embedding optimization}$. The first regularizes the text embedding matrix and effectively suppresses the undesired content. The second method aims to further suppress the unwanted content generation of the prompt, and encourages the generation of desired content. We evaluate our method quantitatively and qualitatively on extensive experiments, validating its effectiveness. Furthermore, our method is generalizability to both the pixel-space diffusion models (i.e. DeepFloyd-IF) and the latent-space diffusion models (i.e. Stable Diffusion).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。