arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.02423cs.LG
收藏

利用对比学习改进文本到图像的合成

Improving Text-to-Image Synthesis Using Contrastive Learning

Georgia State University(佐治亚州立大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, Shihao Ji

摘要:

文本到图像合成的目标是生成与给定文本描述匹配的视觉上逼真的图像。在实际应用中,人类对同一幅图像标注的字幕在内容和词语的选择方面具有很大的差异。同一图像的字幕之间的语言差异导致合成图像偏离地面真实。为了解决这个问题,我们提出了一种对比学习方法来提高合成图像的质量并增强其语义一致性。在预训练阶段,我们利用对比学习的方法来学习一致的文本表示对应于同一图像的字幕。此外,在接下来的GAN训练阶段,我们采用对比学习方法来增强从与同一图像相关的字幕生成的图像之间的一致性。我们评估我们的方法在两个流行的文本到图像的合成模型,AttnGAN和DM-GAN,分别在数据集CUB和COCO。实验结果表明,我们的方法可以有效地提高合成图像的质量方面的三个指标:IS,FID和R-精度。特别是,在具有挑战性的COCO数据集上,我们的方法比AttnGAN显著提高了FID 29.60%,比DM-GAN提高了21.96%。

英文摘要:

The goal of text-to-image synthesis is to generate a visually realistic image that matches a given text description. In practice, the captions annotated by humans for the same image have large variance in terms of contents and the choice of words. The linguistic discrepancy between the captions of the identical image leads to the synthetic images deviating from the ground truth. To address this issue, we propose a contrastive learning approach to improve the quality and enhance the semantic consistency of synthetic images. In the pretraining stage, we utilize the contrastive learning approach to learn the consistent textual representations for the captions corresponding to the same image. Furthermore, in the following stage of GAN training, we employ the contrastive learning method to enhance the consistency between the generated images from the captions related to the same image. We evaluate our approach over two popular text-to-image synthesis models, AttnGAN and DM-GAN, on datasets CUB and COCO, respectively. Experimental results have shown that our approach can effectively improve the quality of synthetic images in terms of three metrics: IS, FID and R-precision. Especially, on the challenging COCO dataset, our approach boosts the FID signifcantly by 29.60% over AttnGAN and by 21.96% over DM-GAN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑