交叉标题：扩展的MS-COCO模态内和模态间语义相似性判断
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
摘要：
通过支持多模态检索训练和评价，图像描述数据集在表示学习方面取得了显著进展。不幸的是，数据集具有有限的跨模态关联：图像不与其他图像配对，标题仅与同一图像的其他标题配对，没有负面关联，并且缺少正面的跨模态关联。这破坏了跨通道学习如何影响通道内任务的研究。我们用交叉字幕（CxC）解决了这一差距，CxC是MS-COCO数据集的扩展，具有267，095个模态内和模态间对的人类语义相似性判断。我们报告CxC的基线结果为强大的现有单峰和多峰模型。我们还评估了一个多任务双编码器训练的图像标题和标题标题对，至关重要的是证明CxC的价值，测量内和间模态学习的影响。
英文摘要：
By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning. Unfortunately, datasets have limited cross-modal associations: images are not paired with other images, captions are only paired with other captions of the same image, there are no negative associations and there are missing positive cross-modal associations. This undermines research into how inter-modality learning impacts intra-modality tasks. We address this gap with Crisscrossed Captions (CxC), an extension of the MS-COCO dataset with human semantic similarity judgments for 267,095 intra- and inter-modality pairs. We report baseline results on CxC for strong existing unimodal and multimodal models. We also evaluate a multitask dual encoder trained on both image-caption and caption-caption pairs that crucially demonstrates CxC's value for measuring the influence of intra- and inter-modality learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
