UM-Text: 一种统一的多模态模型用于图像理解和视觉文本编辑
UM-Text: A Unified Multimodal Model for Image Understanding and Visual Text Editing
AI总结：
本文提出UM-Text模型，通过自然语言指令实现图像理解和视觉文本编辑，引入VLM处理指令和参考图像，结合UM-Encoder生成风格一致的文本图像，并提出区域一致性损失和三阶段训练策略，最终在多个基准上取得最佳性能。
摘要：
随着图像生成技术的快速发展，利用自然语言指令进行视觉文本编辑的研究日益受到关注。该任务的主要挑战在于充分理解指令和参考图像，从而生成与图像风格一致的视觉文本。先前的方法通常涉及复杂步骤来指定文本内容和属性，如字体大小、颜色和布局，而未考虑与参考图像的风格一致性。为了解决这一问题，我们提出了UM-Text，一种统一的多模态模型，用于通过自然语言指令进行上下文理解和视觉文本编辑。具体而言，我们引入了视觉语言模型（VLM）来处理指令和参考图像，从而根据上下文信息精心设计文本内容和布局。为了生成准确且和谐的视觉文本图像，我们进一步提出了UM-Encoder，用于结合各种条件信息的嵌入，其中组合方式由VLM根据输入指令自动配置。在训练过程中，我们提出了一种区域一致性损失，以在潜在空间和RGB空间中为Glyph生成提供更有效的监督，并设计了一种定制的三阶段训练策略，以进一步提升模型性能。此外，我们还贡献了UM-DATA-200K，一个大规模的视觉文本图像数据集，涵盖多样的场景用于模型训练。在多个公开基准上的大量定性和定量结果表明，我们的方法在多个基准上取得了最先进的性能。
英文摘要：
With the rapid advancement of image generation, visual text editing using natural language instructions has received increasing attention. The main challenge of this task is to fully understand the instruction and reference image, and thus generate visual text that is style-consistent with the image. Previous methods often involve complex steps of specifying the text content and attributes, such as font size, color, and layout, without considering the stylistic consistency with the reference image. To address this, we propose UM-Text, a unified multimodal model for context understanding and visual text editing by natural language instructions. Specifically, we introduce a Visual Language Model (VLM) to process the instruction and reference image, so that the text content and layout can be elaborately designed according to the context information. To generate an accurate and harmonious visual text image, we further propose the UM-Encoder to combine the embeddings of various condition information, where the combination is automatically configured by VLM according to the input instruction. During training, we propose a regional consistency loss to offer more effective supervision for glyph generation on both latent and RGB space, and design a tailored three-stage training strategy to further enhance model performance. In addition, we contribute the UM-DATA-200K, a large-scale visual text image dataset on diverse scenes for model training. Extensive qualitative and quantitative results on multiple public benchmarks demonstrate that our method achieves state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
