利用双编码器改进面向场景文本编辑的扩散模型
Improving Diffusion Models for Scene Text Editing with Dual Encoders
AI总结：
本文提出DIFFSTE，通过字符编码器和指令编码器改进预训练扩散模型，提升场景文本编辑中的文本可读性与风格控制能力，并实现字体变化、字体混合及自然语言指令的零样本泛化。
摘要：
场景文本编辑是一项具有挑战性的任务，涉及在图像中修改或插入指定文本，同时保持其自然逼真的外观。此前大多数方法依赖风格迁移模型，这些模型裁剪出文本区域并将其输入图像迁移模型（如GAN）。然而，这些方法在改变文本风格方面能力有限，且无法在图像中插入文本。扩散模型的最新进展显示出通过文本条件图像编辑克服这些局限的潜力。然而，我们的实证分析表明，最先进的扩散模型在渲染正确文本和控制文本风格方面仍存在困难。为解决这些问题，我们提出DIFFSTE，通过双编码器设计改进预训练扩散模型，其中包括一个字符编码器以提高文本可读性，以及一个指令编码器以实现更好的风格控制。我们引入一个指令微调框架来训练模型学习从文本指令到相应图像的映射，该图像可以具有指定风格或背景中周围文本的风格。这种训练方法进一步赋予我们的方法对以下三种场景的零样本泛化能力：生成具有未见字体变化的文本（例如斜体和粗体）、混合不同字体以构建新字体，以及使用更宽松的自然语言形式作为指令来引导生成任务。我们在五个数据集上评估了我们的方法，并证明了其在文本正确性、图像自然度和风格可控性方面的优越性能。我们的代码已公开。https://github.com/UCSB-NLP-Chang/DiffSTE
英文摘要：
Scene text editing is a challenging task that involves modifying or inserting specified texts in an image while maintaining its natural and realistic appearance. Most previous approaches to this task rely on style-transfer models that crop out text regions and feed them into image transfer models, such as GANs. However, these methods are limited in their ability to change text style and are unable to insert texts into images. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. However, our empirical analysis reveals that state-of-the-art diffusion models struggle with rendering correct text and controlling text style. To address these problems, we propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design, which includes a character encoder for better text legibility and an instruction encoder for better style control. An instruction tuning framework is introduced to train our model to learn the mapping from the text instruction to the corresponding image with either the specified style or the style of the surrounding texts in the background. Such a training method further brings our method the zero-shot generalization ability to the following three scenarios: generating text with unseen font variation, e.g., italic and bold, mixing different fonts to construct a new font, and using more relaxed forms of natural language as the instructions to guide the generation task. We evaluate our approach on five datasets and demonstrate its superior performance in terms of text correctness, image naturalness, and style controllability. Our code is publicly available. https://github.com/UCSB-NLP-Chang/DiffSTE
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
