arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.01976cs.CV
收藏

跨模态系统利用语义关系吗?

Do Cross Modal Systems Leverage Semantic Relationships?

University of Insubria(因苏布里亚大学);National University of Sciences and Technology(国立科技大学);Information Technology University(信息技术大学)

Shah Nawaz, Muhammad Kamran Janjua, Ignazio Gallo, Arif Mahmood, Alessandro Calefati, Faisal Shafait

摘要:

目前的跨模态检索系统都是使用R@K测度进行评价的,该测度不利用语义关系,而是严格遵循人工标记的图像文本查询对。因此,目前的系统不能很好地概括野外看不见的数据。为了解决这个问题,我们提出了一个新的措施,SemanticMap,来评估跨模态系统的性能。我们提出的措施评估潜在的嵌入空间中的图像和文本表示之间的语义相似性。我们还提出了一种新型的使用单流网络进行双向检索的跨模态检索系统。所提出的系统是基于一个深度神经网络训练使用扩展的中心损失,最大限度地减少图像和文本描述的距离在潜在的空间从类中心。在我们的系统中,文本描述也被编码为图像,这使我们能够对文本和图像使用单流网络。据我们所知,我们的工作是第一次采用单流网络的跨模态检索系统。建议的系统进行评估,包括MSCOCO和Flickr 30 K两个公开的数据集,并已显示出可比的结果,目前国家的最先进的方法。

英文摘要:

Current cross-modal retrieval systems are evaluated using R@K measure which does not leverage semantic relationships rather strictly follows the manually marked image text query pairs. Therefore, current systems do not generalize well for the unseen data in the wild. To handle this, we propose a new measure, SemanticMap, to evaluate the performance of cross-modal systems. Our proposed measure evaluates the semantic similarity between the image and text representations in the latent embedding space. We also propose a novel cross-modal retrieval system using a single stream network for bidirectional retrieval. The proposed system is based on a deep neural network trained using extended center loss, minimizing the distance of image and text descriptions in the latent space from the class centers. In our system, the text descriptions are also encoded as images which enabled us to use a single stream network for both text and images. To the best of our knowledge, our work is the first of its kind in terms of employing a single stream network for cross-modal retrieval systems. The proposed system is evaluated on two publicly available datasets including MSCOCO and Flickr30K and has shown comparable results to the current state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑