UDiffText：基于字符感知扩散模型的任意图像高质量文本合成统一框架
UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models
AI总结：
本文提出UDiffText，一个基于字符感知扩散模型的统一框架，通过轻量级字符级文本编码器和局部注意力控制，解决文本图像生成中的拼写错误问题，实现任意图像中高质量文本合成。
摘要：
基于扩散模型的文本到图像（T2I）生成方法在过去几年中引起了广泛关注。尽管这些图像合成方法能够产生视觉上吸引人的结果，但在生成图像中渲染文本时，它们经常出现拼写错误。这些错误表现为字符缺失、错误或多余，从而严重限制了基于扩散模型的文本图像生成的性能。为了解决上述问题，本文提出了一种新颖的文本图像生成方法，利用预训练的扩散模型（即Stable Diffusion [27]）。我们的方法涉及设计并训练一个轻量级的字符级文本编码器，该编码器替代了原始的CLIP编码器，并提供更鲁棒的文本嵌入作为条件引导。然后，我们使用大规模数据集对扩散模型进行微调，在字符级分割图的监督下引入局部注意力控制。最后，通过采用推理阶段的细化过程，我们在任意给定图像中合成文本时实现了显著高的序列准确率。定性和定量结果均证明了我们的方法相对于现有技术的优越性。此外，我们展示了所提出的UDiffText的几个潜在应用，包括以文本为中心的图像合成、场景文本编辑等。代码和模型将在https://github.com/ZYM-PKU/UDiffText上提供。
英文摘要：
Text-to-Image (T2I) generation methods based on diffusion model have garnered significant attention in the last few years. Although these image synthesis methods produce visually appealing results, they frequently exhibit spelling errors when rendering text within the generated images. Such errors manifest as missing, incorrect or extraneous characters, thereby severely constraining the performance of text image generation based on diffusion models. To address the aforementioned issue, this paper proposes a novel approach for text image generation, utilizing a pre-trained diffusion model (i.e., Stable Diffusion [27]). Our approach involves the design and training of a light-weight character-level text encoder, which replaces the original CLIP encoder and provides more robust text embeddings as conditional guidance. Then, we fine-tune the diffusion model using a large-scale dataset, incorporating local attention control under the supervision of character-level segmentation maps. Finally, by employing an inference stage refinement process, we achieve a notably high sequence accuracy when synthesizing text in arbitrarily given images. Both qualitative and quantitative results demonstrate the superiority of our method to the state of the art. Furthermore, we showcase several potential applications of the proposed UDiffText, including text-centric image synthesis, scene text editing, etc. Code and model will be available at https://github.com/ZYM-PKU/UDiffText .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
