arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.14124cs.CVcs.AI
收藏

Lafite2:Few-Shot文本到图像生成

Lafite2: Few-shot Text-to-Image Generation

State University of New York at Buffalo(纽约州立大学布法罗分校);Microsoft(微软)

Yufan Zhou, Chunyuan Li, Changyou Chen, Jianfeng Gao, Jinhui Xu

摘要:

文本到图像生成模型近年来取得了很大进展,现在可以从任意文本生成令人印象深刻的逼真图像。大多数这样的模型都是在网络规模的图像-文本配对数据集上训练的,这对许多研究人员来说可能负担不起。在本文中,我们提出了一种新的方法,用于在纯图像数据集上预训练文本到图像生成模型。它考虑了一个检索然后优化过程来合成伪文本特征:对于给定的图像,首先检索相关的伪文本特征,然后优化以更好地对齐。所提出的方法的低要求产生了高灵活性和可用性:它可以有益于广泛的设置,包括Few-Shot、半监督和全监督学习;它可以应用于不同的模型,包括生成对抗网络(GANs)和扩散模型。大量实验证明了所提出方法的有效性。在MS-COCO数据集上,我们的GAN模型获得了6.78的Fréchet Inception Distance(FID),这是全监督设置下GANs的最新技术水平(SoTA)。我们的扩散模型在zero-shot和监督设置上分别获得了8.42和4.28的FID,这与模型尺寸小得多的SoTA扩散模型相比具有竞争力。

英文摘要:

Text-to-image generation models have progressed considerably in recent years, which can now generate impressive realistic images from arbitrary text. Most of such models are trained on web-scale image-text paired datasets, which may not be affordable for many researchers. In this paper, we propose a novel method for pre-training text-to-image generation model on image-only datasets. It considers a retrieval-then-optimization procedure to synthesize pseudo text features: for a given image, relevant pseudo text features are first retrieved, then optimized for better alignment. The low requirement of the proposed method yields high flexibility and usability: it can be beneficial to a wide range of settings, including the few-shot, semi-supervised and fully-supervised learning; it can be applied on different models including generative adversarial networks (GANs) and diffusion models. Extensive experiments illustrate the effectiveness of the proposed method. On MS-COCO dataset, our GAN model obtains Fréchet Inception Distance (FID) of 6.78 which is the new state-of-the-art (SoTA) of GANs under fully-supervised setting. Our diffusion model obtains FID of 8.42 and 4.28 on zero-shot and supervised setting respectively, which are competitive to SoTA diffusion models with a much smaller model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑