arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.12914cs.CV
收藏

基于大语言模型文本表示的文生图生成的实证研究与分析

An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representation

Fudan University(复旦大学);INF Tech(INF科技);Soochow University(苏州大学);Carnegie Mellon University(卡内基梅隆大学)

Zhiyu Tan, Mengping Yang, Luozheng Qin, Hao Yang, Ye Qian, Qiang Zhou, Cheng Zhang, Hao Li

AI总结:

本文提出一种三阶段训练流程及轻量级适配器,将大语言模型作为文本编码器集成到文生图模型中,实现多语言和长上下文支持,提升图像生成质量。

摘要:

忠实文生图生成的一个关键先决条件是准确理解文本输入。现有方法利用CLIP模型的文本编码器来表示输入提示。然而,预训练的CLIP模型仅能编码最多77个token长度的英文。此外,与大型语言模型(LLMs)相比,CLIP的文本编码器模型容量相对有限,LLMs提供多语言输入、支持更长上下文,并实现更优的文本表示。在本文中,我们研究将LLMs作为文本编码器,以提高文生图生成中的语言理解能力。不幸的是,从头开始使用LLMs训练文生图生成模型需要大量的计算资源和数据。为此,我们引入了一个三阶段训练流程,该流程有效且高效地将现有文生图模型与LLMs集成。具体而言,我们提出了一个轻量级适配器,使得能够利用来自LLMs的文本表示快速训练文生图模型。大量实验表明,我们的模型不仅支持多语言,还支持更长的输入上下文,并具有更优的图像生成质量。

英文摘要:

One critical prerequisite for faithful text-to-image generation is the accurate understanding of text inputs. Existing methods leverage the text encoder of the CLIP model to represent input prompts. However, the pre-trained CLIP model can merely encode English with a maximum token length of 77. Moreover, the model capacity of the text encoder from CLIP is relatively limited compared to Large Language Models (LLMs), which offer multilingual input, accommodate longer context, and achieve superior text representation. In this paper, we investigate LLMs as the text encoder to improve the language understanding in text-to-image generation. Unfortunately, training text-to-image generative model with LLMs from scratch demands significant computational resources and data. To this end, we introduce a three-stage training pipeline that effectively and efficiently integrates the existing text-to-image model with LLMs. Specifically, we propose a lightweight adapter that enables fast training of the text-to-image model using the textual representations from LLMs. Extensive experiments demonstrate that our model supports not only multilingual but also longer input context with superior image generation quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑