AnyText：多语言视觉文本生成与编辑
AnyText: Multilingual Visual Text Generation And Editing
AI总结：
AnyText 提出基于扩散模型的多语言视觉文本生成与编辑方法，通过辅助潜在模块和文本嵌入模块实现图像中准确连贯的多语言文本渲染，并贡献了首个大规模多语言文本图像数据集 AnyWord-3M 及评测基准。
摘要：
基于扩散模型的文本到图像（Text-to-Image）技术近年来取得了令人瞩目的成就。尽管当前的图像合成技术已高度先进，能够生成高保真度的图像，但当聚焦于生成图像中的文本区域时，仍可能露出破绽。为了解决这一问题，我们提出了 AnyText，一个基于扩散的多语言视觉文本生成与编辑模型，专注于在图像中渲染准确且连贯的文本。AnyText 包含一个具有两个核心元素的扩散流程：一个辅助潜在模块和一个文本嵌入模块。前者利用文本字形、位置和掩码图像等输入来生成用于文本生成或编辑的潜在特征。后者采用 OCR 模型将笔画数据编码为嵌入，这些嵌入与来自分词器的图像描述嵌入相融合，从而生成与背景无缝融合的文本。我们在训练中采用了文本控制扩散损失和文本感知损失，以进一步提升书写准确性。AnyText 能够书写多种语言的字符，据我们所知，这是首个解决多语言视觉文本生成问题的工作。值得一提的是，AnyText 可以插入社区现有的扩散模型中，以准确地渲染或编辑文本。经过广泛的评估实验，我们的方法以显著优势超越了所有其他方法。此外，我们贡献了首个大规模多语言文本图像数据集 AnyWord-3M，包含 300 万个带有多种语言 OCR 标注的图像-文本对。基于 AnyWord-3M 数据集，我们提出了 AnyText-benchmark，用于评估视觉文本生成的准确性和质量。我们的项目将在 https://github.com/tyxsspa/AnyText 上开源，以改进和推动文本生成技术的发展。
英文摘要：
Diffusion model based Text-to-Image has achieved impressive achievements recently. Although current technology for synthesizing images is highly advanced and capable of generating images with high fidelity, it is still possible to give the show away when focusing on the text area in the generated image. To address this issue, we introduce AnyText, a diffusion-based multilingual visual text generation and editing model, that focuses on rendering accurate and coherent text in the image. AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. AnyText can write characters in multiple languages, to the best of our knowledge, this is the first work to address multilingual visual text generation. It is worth mentioning that AnyText can be plugged into existing diffusion models from the community for rendering or editing text accurately. After conducting extensive evaluation experiments, our method has outperformed all other approaches by a significant margin. Additionally, we contribute the first large-scale multilingual text images dataset, AnyWord-3M, containing 3 million image-text pairs with OCR annotations in multiple languages. Based on AnyWord-3M dataset, we propose AnyText-benchmark for the evaluation of visual text generation accuracy and quality. Our project will be open-sourced on https://github.com/tyxsspa/AnyText to improve and promote the development of text generation technology.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
