arXiv 2306.142592023-06-25cs.CV

利用对比奖励改进基于参考的区分性图像描述

Improving Reference-based Distinctive Image Captioning with Contrastive Rewards

Zhejiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）；Peking University（北京大学）

Yangjun Mao, Jun Xiao, Dong Zhang, Meng Cao, Jian Shao, Yueting Zhuang, Long Chen

备注：: arXiv admin note: substantial text overlap with arXiv:2207.11118

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究提出两个新Ref-DIC基准和基于Transformer的TransDIC模型，并进一步提出包含对比学习模块的TransDIC++和新评估指标DisCIDEr，在多个指标上超越现有模型。

摘要：

区分性图像描述（DIC）——生成描述目标图像独特细节的区分性描述——在过去几年中受到了广泛关注。最近的一种DIC方法提出通过将目标图像与一组语义相似的参考图像进行比较来生成区分性描述，即基于参考的DIC（Ref-DIC）。它旨在迫使生成的描述区分目标图像和参考图像。为了确保Ref-DIC模型真正感知目标图像中的独特对象（或属性），我们提出了两个新的Ref-DIC基准，并开发了一个基于Transformer的Ref-DIC基线模型TransDIC。该模型不仅从目标图像中提取视觉特征，还对目标图像和参考图像中对象之间的差异进行编码。。更进一步，我们提出了一个更强大的TransDIC++，它由一个额外的对比学习模块组成，以充分利用参考图像。这个新模块是与模型无关的，可以轻松地整合到各种Ref-DIC架构中。最后，为了实现更可信的基准测试，我们为Ref-DIC提出了一种名为DisCIDEr的新评估指标，该指标评估生成描述的准确性和区分性。实验结果表明，我们的TransDIC++能够生成区分性描述。此外，它在两个新基准的不同指标上均优于几个最先进的模型。

英文摘要：

Distinctive Image Captioning (DIC) -- generating distinctive captions that describe the unique details of a target image -- has received considerable attention over the last few years. A recent DIC method proposes to generate distinctive captions by comparing the target image with a set of semantic-similar reference images, i.e., reference-based DIC (Ref-DIC). It aims to force the generated captions to distinguish between the target image and the reference image. To ensure Ref-DIC models really perceive the unique objects (or attributes) in target images, we propose two new Ref-DIC benchmarks and develop a Transformer-based Ref-DIC baseline TransDIC. The model only extracts visual features from the target image, but also encodes the differences between objects in the target and reference images. Taking one step further, we propose a stronger TransDIC++, which consists of an extra contrastive learning module to make full use of the reference images. This new module is model-agnostic, which can be easily incorporated into various Ref-DIC architectures. Finally, for more trustworthy benchmarking, we propose a new evaluation metric named DisCIDEr for Ref-DIC, which evaluates both the accuracy and distinctiveness of the generated captions. Experimental results demonstrate that our TransDIC++ can generate distinctive captions. Besides, it outperforms several state-of-the-art models on the two new benchmarks over different metrics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。