arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.03045cs.CV
收藏

面向文本到图像生成的定制助手

Customization Assistant for Text-to-image Generation

Adobe Research（奥多比研究院）

Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun

备注：
CVPR 2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于预训练大语言模型与扩散模型的文本到图像定制助手，通过新模型设计和训练策略实现免微调、2至5秒生成及自然语言交互。

摘要：

定制预训练的文本到图像生成模型，由于其在现实应用中的巨大潜力，近来吸引了大量研究兴趣。尽管现有方法能够针对单张用户输入图像中包含的新颖概念生成富有创意的内容，但其能力仍远未达到完美。具体而言，大多数现有方法需要在测试图像上微调生成模型。一些现有方法不需要微调，但其性能却不能令人满意。此外，用户与模型之间的交互仍局限于指令和描述性提示，例如操作说明和图像说明。在本工作中，我们基于预训练的大语言模型和扩散模型构建了一个定制助手，它不仅能够以免微调的方式执行定制化生成，还支持更加用户友好的交互：用户可以与助手聊天，并输入模糊文本或清晰指令。具体来说，我们提出了一个由新模型设计和新颖训练策略组成的新框架。所得助手无需任何测试时微调，即可在2至5秒内完成定制化生成。我们开展了大量实验，并在不同领域取得了具有竞争力的结果，证明了所提方法的有效性。

英文摘要：

Customizing pre-trained text-to-image generation model has attracted massive research interest recently, due to its huge potential in real-world applications. Although existing methods are able to generate creative content for a novel concept contained in single user-input image, their capability are still far from perfection. Specifically, most existing methods require fine-tuning the generative model on testing images. Some existing methods do not require fine-tuning, while their performance are unsatisfactory. Furthermore, the interaction between users and models are still limited to directive and descriptive prompts such as instructions and captions. In this work, we build a customization assistant based on pre-trained large language model and diffusion model, which can not only perform customized generation in a tuning-free manner, but also enable more user-friendly interactions: users can chat with the assistant and input either ambiguous text or clear instruction. Specifically, we propose a new framework consists of a new model design and a novel training strategy. The resulting assistant can perform customized generation in 2-5 seconds without any test time fine-tuning. Extensive experiments are conducted, competitive results have been obtained across different domains, illustrating the effectiveness of the proposed method.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑