arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.01855cs.CVcs.AI
收藏

使用扩散模型进行图像描述的多模态数据增强

Multimodal Data Augmentation for Image Captioning using Diffusion Models

School of Economics and Management, Tsinghua University(清华大学经济管理学院);Department of Decision, Operations & Information Technologies, University of Maryland(马里兰大学决策、运营与信息技术系)

Changrong Xiao, Sean Xin Xu, Kunpeng Zhang

AI总结:

本文提出利用Stable Diffusion生成图像-描述对进行多模态数据增强,在MS COCO上显著提升图像描述性能,尤其在小样本场景下效果突出。

摘要:

图像描述是一项重要的视觉语言任务,通常需要大量精细标注的图像-描述对来学习图像与文本之间的潜在对齐关系。本文提出了一种多模态数据增强方法,利用近期提出的文本到图像模型Stable Diffusion,通过高质量生成图像-描述对来扩充训练集。在MS COCO数据集上的大量实验证明了我们的方法相较于若干基准方法的优势,尤其是在训练样本较少时提升显著。此外,基于我们增强数据集训练的模型也大幅超越了先前的无配对图像描述方法。最后,在根据质量评估对生成数据进行有意筛选后,训练效率和效果可得到进一步提升。

英文摘要:

Image captioning, an important vision-language task, often requires a tremendous number of finely labeled image-caption pairs for learning the underlying alignment between images and texts. In this paper, we proposed a multimodal data augmentation method, leveraging a recent text-to-image model called Stable Diffusion, to expand the training set via high-quality generation of image-caption pairs. Extensive experiments on the MS COCO dataset demonstrate the advantages of our approach over several benchmark methods, and particularly a significant boost when having fewer training instances. In addition, models trained on our augmented datasets also outperform prior unpaired image captioning methods by a large margin. Finally, further improvement regarding the training efficiency and effectiveness can be obtained after intentionally filtering the generated data based on quality assessment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑