arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.03400cs.CV
收藏

通过定向 Token 调整编辑 Stable Diffusion 中的隐式先验

Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustment

University of Sheffield（谢菲尔德大学）；Toshiba Europe Ltd（东芝欧洲有限公司）

Feng He, Chao Zhang, Zhixue Zhao

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 Embedit，仅微调目标对象的词元嵌入以定向修改 Stable Diffusion 的隐式先验，单次编辑改动极少且可逆，并在多场景中较已有方法至少提升 6.01%。

摘要：

在文本到图像生成任务中，隐式假设和先验通常是必要的，尤其是在文本提示缺乏足够上下文时。然而，这些假设有时会反映训练数据中嵌入的过时概念、不准确之处或社会偏见。我们提出 Embedding-only Editing（Embedit），这是一种旨在高效调整模型中隐式假设和先验、同时不影响其对无关对象的解释或整体性能的方法。给定一个会引发隐式假设（例如玫瑰是红色的）的“源”提示（例如“rose”），以及一个指定期望属性（例如“blue rose”）的“目标”提示，Embedit 仅微调目标对象（“rose”）的词元嵌入（word token embedding，WTE），以优化 SOTA 文本到图像模型 Stable Diffusion 中文本编码器的最后一个隐藏状态。这种定向调整可防止对模型知识库中其他对象产生意外影响，因为无关对象的 WTE 和模型权重保持不变。因此，当提示不包含被编辑对象时，所有表示和模型输出都与原始的、未经编辑的模型完全相同。我们的方法非常高效，单次编辑仅修改 Stable Diffusion 1.4 的 768 个参数和 XL 的 2048 个参数，与各自模型的 WTE 维度相匹配。这种极小的修改范围加上快速执行，使 Embedit 在实际应用中极具实用性。此外，通过恢复原始 WTE 层即可轻松撤销更改。实验结果表明，Embedit 在各种模型、任务和编辑场景（单次编辑和连续多次编辑）中始终优于以往方法，实现了至少 6.01% 的提升（从 87.17% 提升至 93.18%）。

英文摘要：

Implicit assumptions and priors are often necessary in text-to-image generation tasks, especially when textual prompts lack sufficient context. However, these assumptions can sometimes reflect outdated concepts, inaccuracies, or societal bias embedded in the training data. We present Embedding-only Editing (Embedit), a method designed to efficiently adjust implict assumptions and priors in the model without affecting its interpretation of unrelated objects or overall performance. Given a "source" prompt (e.g., "rose") that elicits an implicit assumption (e.g., rose is red) and a "destination" prompt that specifies the desired attribute (e.g., "blue rose"), Embedit fine-tunes only the word token embedding (WTE) of the target object ("rose") to optimize the last hidden state of text encoder in Stable Diffusion, a SOTA text-to-image model. This targeted adjustment prevents unintended effects on other objects in the model's knowledge base, as the WTEs for unrelated objects and the model weights remain unchanged. Consequently, when a prompt does not contain the edited object, all representations, and the model outputs are identical to those of the original, unedited model. Our method is highly efficient, modifying only 768 parameters for Stable Diffusion 1.4 and 2048 for XL in a single edit, matching the WTE dimension of each respective model. This minimal scope, combined with rapid execution, makes Embedit highly practical for real-world applications. Additionally, changes are easily reversible by restoring the original WTE layers. Our experimental results demonstrate that Embedit consistently outperforms previous methods across various models, tasks, and editing scenarios (both single and sequential multiple edits), achieving at least a 6.01% improvement (from 87.17% to 93.18%).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑