arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1907.09748cs.CLcs.IRcs.LG
收藏

基于位置聚焦注意网络的图文匹配

Position Focused Attention Network for Image-Text Matching

Tencent(腾讯);Xi’an Jiaotong University(西安交通大学);Tencent AI Lab(腾讯人工智能实验室)

Yaxiong Wang, Hao Yang, Xueming Qian, Lin Ma, Jing Lu, Biao Li, Xin Fan

摘要:

图像-文本匹配任务最近在计算机视觉领域引起了很多关注。这种跨领域问题的关键是如何准确地度量视觉内容和文本内容之间的相似性,这需要对视觉和文本两种模态有很好的理解。在本文中,我们提出了一个新的位置集中注意力网络(PFAN)的视觉和文本的意见之间的关系进行调查。在这项工作中,我们整合的对象位置线索,以加强视觉-文本联合嵌入学习。我们首先将图像分割成块,通过块来推断区域在图像中的相对位置。然后,提出一种注意力机制,对图像区域和块之间的关系进行建模,生成有价值的位置特征,进一步利用这些位置特征增强区域表达,并在视觉图像和文本语句之间建立更可靠的关系。在Flickr 30 K和MS-COCO数据集上的实验表明了该方法的有效性。除公开数据集外,本文还在实际的大规模新闻数据集(Criminent-News)上进行了实验,验证了该方法的实际应用价值。据我们所知,这是第一次尝试在实际应用中测试性能。我们的方法在所有这三个数据集上都达到了最先进的性能。

英文摘要:

Image-text matching tasks have recently attracted a lot of attention in the computer vision field. The key point of this cross-domain problem is how to accurately measure the similarity between the visual and the textual contents, which demands a fine understanding of both modalities. In this paper, we propose a novel position focused attention network (PFAN) to investigate the relation between the visual and the textual views. In this work, we integrate the object position clue to enhance the visual-text joint-embedding learning. We first split the images into blocks, by which we infer the relative position of region in the image. Then, an attention mechanism is proposed to model the relations between the image region and blocks and generate the valuable position feature, which will be further utilized to enhance the region expression and model a more reliable relationship between the visual image and the textual sentence. Experiments on the popular datasets Flickr30K and MS-COCO show the effectiveness of the proposed method. Besides the public datasets, we also conduct experiments on our collected practical large-scale news dataset (Tencent-News) to validate the practical application value of proposed method. As far as we know, this is the first attempt to test the performance on the practical application. Our method achieves the state-of-art performance on all of these three datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑