arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.01645cs.NEcs.AIcs.LG
收藏

通过CLIP引导的生成潜在空间搜索从标题生成图像,反之亦然

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

University of Pisa(比萨大学)

Federico A. Galatolo, Mario G. C. A. Cimino, Gigliola Vaglini

摘要:

在这项研究工作中,我们提出了CLIP-GLaSS,一种新的zero-shot框架,以生成对应于给定的字幕(或图像)的图像(或字幕)。CLIP-GLaSS基于CLIP神经网络,给定图像和描述性标题,CLIP神经网络提供类似的嵌入。CLIP-GLaSS以字幕(或图像)为输入,并生成CLIP嵌入与输入图像最相似的图像(或字幕)。在通过遗传算法进行探索之后,通过生成网络产生该最佳图像(或字幕)。基于图像生成器BigGAN和StyleGAN 2以及文本生成器GPT 2的实验,显示了有希望的结果

英文摘要:

In this research work we present CLIP-GLaSS, a novel zero-shot framework to generate an image (or a caption) corresponding to a given caption (or image). CLIP-GLaSS is based on the CLIP neural network, which, given an image and a descriptive caption, provides similar embeddings. Differently, CLIP-GLaSS takes a caption (or an image) as an input, and generates the image (or the caption) whose CLIP embedding is the most similar to the input one. This optimal image (or caption) is produced via a generative network, after an exploration by a genetic algorithm. Promising results are shown, based on the experimentation of the image Generators BigGAN and StyleGAN2, and of the text Generator GPT2

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑