arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.04439cs.CVcs.AI
收藏

通过输入粒度控制与字形感知训练赋能骨干模型实现视觉文本生成

Empowering Backbone Models for Visual Text Generation with Input Granularity Control and Glyph-Aware Training

School of Informatics, Xiamen University(厦门大学信息学院);Baidu Inc.(百度公司);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Wenbo Li, Guohao Li, Zhibin Lan, Xue Xu, Wanru Zhuang, Jiachen Liu, Xinyan Xiao, Jinsong Su

AI总结:

针对扩散文本生成图像模型难以生成清晰视觉文本的问题,提出混合粒度输入策略与字形感知训练损失,赋能骨干模型生成准确美观的中英文视觉文本图像且保留基础生成质量。

摘要:

基于扩散的文本生成图像模型在多样性和美观性上已取得亮眼成果,但难以生成带有清晰可读视觉文本的图像。现有骨干模型存在拼写错误、无法生成文本、不支持中文文本等局限,但其发展展现出可观潜力。本文提出一系列方法,旨在赋能骨干模型生成中英文视觉文本。我们首先开展了初步研究,发现字节对编码(Byte Pair Encoding, BPE)分词机制以及交叉注意力模块的学习不足限制了骨干模型的性能。基于这些观察,我们做出以下改进:(1)设计混合粒度输入策略,以提供更适配的文本表示;(2)提出在传统训练目标中加入三种字形感知训练损失,强化交叉注意力模块的学习,促使模型关注视觉文本。通过实验我们证明,所提方法能有效赋能骨干模型,生成语义相关、美观且准确的视觉文本图像,同时保留其基础的图像生成质量。

英文摘要:

Diffusion-based text-to-image models have demonstrated impressive achievements in diversity and aesthetics but struggle to generate images with legible visual texts. Existing backbone models have limitations such as misspelling, failing to generate texts, and lack of support for Chinese text, but their development shows promising potential. In this paper, we propose a series of methods, aiming to empower backbone models to generate visual texts in English and Chinese. We first conduct a preliminary study revealing that Byte Pair Encoding (BPE) tokenization and the insufficient learning of cross-attention modules restrict the performance of the backbone models. Based on these observations, we make the following improvements: (1) We design a mixed granularity input strategy to provide more suitable text representations; (2) We propose to augment the conventional training objective with three glyph-aware training losses, which enhance the learning of cross-attention modules and encourage the model to focus on visual texts. Through experiments, we demonstrate that our methods can effectively empower backbone models to generate semantic relevant, aesthetically appealing, and accurate visual text images, while maintaining their fundamental image generation quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑