arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.12249cs.CVcs.LG
收藏

DALL-E 和 Flamingo 能相互理解吗?

Do DALL-E and Flamingo Understand Each Other?

LMU Munich(慕尼黑大学);Siemens AG(西门子公司);University of Oxford(牛津大学)

Hang Li, Jindong Gu, Rajat Koner, Sahand Sharifzadeh, Volker Tresp

AI总结:

本文通过图像描述—图像重建闭环考察图像到文本与文本到图像模型的相互理解,并提出以重建损失联合微调两类模型的统一框架。

摘要:

聚焦图像与文本理解及生成的多模态研究领域已取得显著进展。这一进展的例证是,出现了专门用于大规模图像描述的复杂模型,例如著名的 Flamingo 模型,以及文本到图像生成模型,其中 DALL-E 是一个突出代表。该领域中一个值得探索的有趣问题是:Flamingo 和 DALL-E 是否能够相互理解。为研究这一问题,我们提出了一项重建任务:Flamingo 为给定图像生成一段描述,DALL-E 再以该描述为输入合成一张新图像。我们认为,如果生成图像与给定图像相似,则说明这些模型能够相互理解。具体而言,我们研究了图像重建质量与文本生成质量之间的关系。我们发现,一幅图像的最优描述是能够生成与原始图像相似图像的描述。这一发现促使我们提出一个统一框架,用于微调文本到图像模型和图像到文本模型。具体来说,重建部分构成一个正则化损失,用于指导模型的调优。我们在多个数据集上使用不同的图像描述和图像生成模型进行了大量实验,结果验证了我们的发现,并证明了所提出统一框架的有效性。由于 DALL-E 和 Flamingo 未公开发布,我们在后续工作中使用 Stable Diffusion 和 BLIP。项目网站:https://dalleflamingo.github.io。

英文摘要:

The field of multimodal research focusing on the comprehension and creation of both images and text has witnessed significant strides. This progress is exemplified by the emergence of sophisticated models dedicated to image captioning at scale, such as the notable Flamingo model and text-to-image generative models, with DALL-E serving as a prominent example. An interesting question worth exploring in this domain is whether Flamingo and DALL-E understand each other. To study this question, we propose a reconstruction task where Flamingo generates a description for a given image and DALL-E uses this description as input to synthesize a new image. We argue that these models understand each other if the generated image is similar to the given image. Specifically, we study the relationship between the quality of the image reconstruction and that of the text generation. We find that an optimal description of an image is one that gives rise to a generated image similar to the original one. The finding motivates us to propose a unified framework to finetune the text-to-image and image-to-text models. Concretely, the reconstruction part forms a regularization loss to guide the tuning of the models. Extensive experiments on multiple datasets with different image captioning and image generation models validate our findings and demonstrate the effectiveness of our proposed unified framework. As DALL-E and Flamingo are not publicly available, we use Stable Diffusion and BLIP in the remaining work. Project website: https://dalleflamingo.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑