arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.01645cs.CV
收藏

DesignDiffusion:基于扩散模型的高质量文本到设计图像生成

DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models

University of Science and Technology of China(中国科学技术大学);Microsoft Research Asia(微软亚洲研究院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)

Zhendong Wang, Jianmin Bao, Shuyang Gu, Dong Chen, Wengang Zhou, Houqiang Li

AI总结:

提出DesignDiffusion框架,通过端到端单阶段扩散模型结合字符嵌入、字符定位损失及自博弈DPO微调策略,解决设计图像中文本与视觉元素风格不一致问题,实现高质量文本到设计图像生成。

摘要:

本文提出DesignDiffusion,一个简单而有效的框架,用于从文本描述合成设计图像这一新任务。主要挑战在于生成准确且风格一致的文本与视觉内容。现有视觉文本生成相关工作通常专注于在给定特定区域内生成文本,这限制了生成模型的创造力,若应用于设计图像生成,会导致文本与视觉元素之间的风格或颜色不一致。为解决此问题,我们提出一个端到端、单阶段的基于扩散的框架,避免了位置和布局建模等复杂组件。具体而言,该框架直接从用户提示合成文本和视觉设计元素。它利用源自视觉文本的独特字符嵌入来增强输入提示,并引入字符定位损失以在文本生成期间增强监督。此外,我们采用自博弈Direct Preference Optimization微调策略来提升合成视觉文本的质量和准确性。大量实验表明,DesignDiffusion在设计图像生成中达到了最先进的性能。

英文摘要:

In this paper, we present DesignDiffusion, a simple yet effective framework for the novel task of synthesizing design images from textual descriptions. A primary challenge lies in generating accurate and style-consistent textual and visual content. Existing works in a related task of visual text generation often focus on generating text within given specific regions, which limits the creativity of generation models, resulting in style or color inconsistencies between textual and visual elements if applied to design image generation. To address this issue, we propose an end-to-end, one-stage diffusion-based framework that avoids intricate components like position and layout modeling. Specifically, the proposed framework directly synthesizes textual and visual design elements from user prompts. It utilizes a distinctive character embedding derived from the visual text to enhance the input prompt, along with a character localization loss for enhanced supervision during text generation. Furthermore, we employ a self-play Direct Preference Optimization fine-tuning strategy to improve the quality and accuracy of the synthesized visual text. Extensive experiments demonstrate that DesignDiffusion achieves state-of-the-art performance in design image generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑