Glyph-ByT5：面向精准视觉文本渲染的定制化文本编码器
Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering
AI总结：
针对文本生成图像模型中文本渲染准确率低的问题，研究团队提出定制化字符感知文本编码器Glyph-ByT5，与SDXL集成后大幅提升文本渲染精度，还支持段落渲染与开放域场景文本渲染。
摘要：
视觉文本渲染是当前文本生成图像模型面临的一项核心挑战，其核心问题在于文本编码器的缺陷。为实现精准的文本渲染，我们明确了文本编码器的两项关键要求：字符感知能力以及与字形的对齐能力。我们的解决方案是构建一系列定制化文本编码器Glyph-ByT5，具体通过使用精心策划的字形-文本配对数据集，对具备字符感知能力的ByT5编码器进行微调得到。我们提出了一种将Glyph-ByT5与SDXL集成的有效方法，由此构建出用于设计图像生成的Glyph-SDXL模型。这一改进大幅提升了文本渲染准确率，在我们的设计图像基准测试中，准确率从不足20%提升至接近90%。值得注意的是，Glyph-SDXL新增了文本段落渲染能力，可通过自动多行布局，实现数十到数百个字符的高拼写准确率渲染。最后，通过使用一小部分带有视觉文本的高质量写实图像对Glyph-SDXL进行微调，我们证明了该模型在开放域真实图像中的场景文本渲染能力得到了显著提升。这些颇具说服力的结果旨在推动学界进一步探索为各类复杂任务设计定制化文本编码器的方向。
英文摘要：
Visual text rendering poses a fundamental challenge for contemporary text-to-image generation models, with the core problem lying in text encoder deficiencies. To achieve accurate text rendering, we identify two crucial requirements for text encoders: character awareness and alignment with glyphs. Our solution involves crafting a series of customized text encoder, Glyph-ByT5, by fine-tuning the character-aware ByT5 encoder using a meticulously curated paired glyph-text dataset. We present an effective method for integrating Glyph-ByT5 with SDXL, resulting in the creation of the Glyph-SDXL model for design image generation. This significantly enhances text rendering accuracy, improving it from less than $20\%$ to nearly $90\%$ on our design image benchmark. Noteworthy is Glyph-SDXL's newfound ability for text paragraph rendering, achieving high spelling accuracy for tens to hundreds of characters with automated multi-line layouts. Finally, through fine-tuning Glyph-SDXL with a small set of high-quality, photorealistic images featuring visual text, we showcase a substantial improvement in scene text rendering capabilities in open-domain real images. These compelling outcomes aim to encourage further exploration in designing customized text encoders for diverse and challenging tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
