arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.13792cs.CVcs.LG
收藏

LAFITE:面向文本到图像生成的无障碍训练

LAFITE: Towards Language-Free Training for Text-to-Image Generation

State University of New York at Buffalo(纽约州立大学布法罗分校);Adobe Research(Adobe研究院);Microsoft Research(微软研究院)

Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, Tong Sun

摘要:

训练文本到图像生成模型的主要挑战之一是需要大量高质量的图像-文本对。虽然图像样本通常很容易访问,但相关的文本描述通常需要仔细的人工字幕,这特别耗时和成本。在本文中,我们提出了第一个工作,训练文本到图像生成模型,没有任何文本数据。我们的方法利用了强大的预训练CLIP模型的良好对齐的多模态语义空间:通过从图像特征生成文本特征,无缝地减轻了文本条件的要求。大量的实验表明了该方法的有效性。我们在标准的文本到图像生成任务中获得了最先进的结果。重要的是,所提出的无语言模型优于大多数使用完整图像-文本对训练的现有模型。此外,我们的方法可以应用于微调预训练模型,这节省了训练时间和训练文本到图像生成模型的成本。我们的预训练模型在MS-COCO数据集上的zero-shot文本到图像生成中获得了有竞争力的结果,但相对于最近提出的大型DALL-E模型,模型大小和训练数据大小仅为1%左右。

英文摘要:

One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs. While image samples are often easily accessible, the associated text descriptions typically require careful human captioning, which is particularly time- and cost-consuming. In this paper, we propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multi-modal semantic space of the powerful pre-trained CLIP model: the requirement of text-conditioning is seamlessly alleviated via generating text features from image features. Extensive experiments are conducted to illustrate the effectiveness of the proposed method. We obtain state-of-the-art results in the standard text-to-image generation tasks. Importantly, the proposed language-free model outperforms most existing models trained with full image-text pairs. Furthermore, our method can be applied in fine-tuning pre-trained models, which saves both training time and cost in training text-to-image generation models. Our pre-trained model obtains competitive results in zero-shot text-to-image generation on the MS-COCO dataset, yet with around only 1% of the model size and training data size relative to the recently proposed large DALL-E model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑