Prompt-Softbox-Prompt：一种用于图像编辑的自由文本嵌入控制方法
Prompt-Softbox-Prompt: A Free-Text Embedding Control for Image Editing
AI总结：
本文分析了Stable Diffusion XL文本嵌入特性，提出无需训练的PSP图像编辑方法，通过修改交叉注意力层文本嵌入并用Softbox控制语义注入区域，实现精确的对象替换、添加及风格迁移。
摘要：
尽管文本驱动的扩散模型在图像编辑中表现出色，但其文本嵌入的关键组件仍未被充分探索。这些嵌入的模糊性和纠缠性给精确编辑带来了挑战。本文对Stable Diffusion XL中的文本嵌入进行了全面分析，提供了三个关键见解：(1) aug embedding（增强嵌入）保留了完整的文本语义，但由于仅通过ResBlocks融合，对图像生成的贡献极小。更多的文本信息会削弱其局部语义，同时保留大部分全局语义。(2) BOS和padding embedding（填充嵌入）不包含任何语义信息。(3) EOS包含所有单词的语义信息和风格信息。每个单词嵌入都很重要，且不会干扰其他嵌入的语义注入。基于这些见解，我们提出了PSP（Prompt-Softbox-Prompt），一种无需训练的图像编辑方法，利用自由文本嵌入。PSP通过修改交叉注意力层中的文本嵌入并使用Softbox控制语义注入的特定区域，实现精确的图像编辑。该技术能够在不影响图像其他区域的情况下添加和替换对象。此外，PSP仅需替换文本嵌入即可实现风格迁移。大量实验表明，PSP在对象替换、对象添加和风格迁移等任务中表现优异。代码可在https://github.com/yangyt46/PSP获取。
英文摘要：
While text-driven diffusion models demonstrate remarkable performance in image editing, the critical components of their text embeddings remain underexplored. The ambiguity and entanglement of these embeddings pose challenges for precise editing. In this paper, we provide a comprehensive analysis of text embeddings in Stable Diffusion XL, offering three key insights: (1) \textit{aug embedding}~\footnote{\textit{aug embedding} is obtained by combining the pooled output of the final text encoder with the timestep embeddings. https://github.com/huggingface/diffusers} retains complete textual semantics but contributes minimally to image generation as it is only fused via the ResBlocks. More text information weakens its local semantics while preserving most global semantics. (2) \textit{BOS} and \textit{padding embedding} do not contain any semantic information. (3) \textit{EOS} holds the semantic information of all words and stylistic information. Each word embedding is important and does not interfere with the semantic injection of other embeddings. Based on these insights, we propose PSP (\textbf{P}rompt-\textbf{S}oftbox-\textbf{P}rompt), a training-free image editing method that leverages free-text embedding. PSP enables precise image editing by modifying text embeddings within the cross-attention layers and using Softbox to control the specific area for semantic injection. This technique enables the addition and replacement of objects without affecting other areas of the image. Additionally, PSP can achieve style transfer by simply replacing text embeddings. Extensive experiments show that PSP performs remarkably well in tasks such as object replacement, object addition, and style transfer. Our code is available at https://github.com/yangyt46/PSP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
