arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1707.09472cs.CV
收藏

视觉关系的弱监督学习

Weakly-supervised learning of visual relations

INRIA(法国国家信息与自动化研究所);Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK(格勒诺布尔阿尔卑斯大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔理工学院、让·昆兹曼实验室);Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague(布拉格捷克理工大学信息学、机器人与控制论研究所)

Julia Peyre, Ivan Laptev, Cordelia Schmid, Josef Sivic

摘要:

本文介绍了一种新的方法来建模对象对之间的视觉关系。我们称关系为三元组的形式(主语,谓语,宾语),其中谓语通常是介词(例如。'under','in front of')或连接一对宾语(主语,宾语)的动词('hold','ride')。学习这种关系是具有挑战性的,因为对象具有不同的空间配置和外观,这取决于它们发生的关系。另一个主要的挑战来自于很难为所有可能的三元组获得注释,特别是在框级,这使得学习和评估都很困难。本文的贡献有三个方面。首先,我们设计了强大而灵活的视觉功能,对成对对象的外观和空间配置进行编码。其次,我们提出了一个弱监督判别聚类模型,只从图像级标签中学习关系。第三,我们引入了一个新的具有挑战性的数据集的不寻常的关系(UnRel)连同详尽的注释,使视觉关系检索的准确评估。我们通过实验证明,我们的模型在视觉关系数据集上产生了最先进的结果,显著提高了以前看不见的关系(零样本学习)的性能,并在我们新引入的UnRel数据集上证实了这一观察结果。

英文摘要:

This paper introduces a novel approach for modeling visual relations between pairs of objects. We call relation a triplet of the form (subject, predicate, object) where the predicate is typically a preposition (eg. 'under', 'in front of') or a verb ('hold', 'ride') that links a pair of objects (subject, object). Learning such relations is challenging as the objects have different spatial configurations and appearances depending on the relation in which they occur. Another major challenge comes from the difficulty to get annotations, especially at box-level, for all possible triplets, which makes both learning and evaluation difficult. The contributions of this paper are threefold. First, we design strong yet flexible visual features that encode the appearance and spatial configuration for pairs of objects. Second, we propose a weakly-supervised discriminative clustering model to learn relations from image-level labels only. Third we introduce a new challenging dataset of unusual relations (UnRel) together with an exhaustive annotation, that enables accurate evaluation of visual relation retrieval. We show experimentally that our model results in state-of-the-art results on the visual relationship dataset significantly improving performance on previously unseen relations (zero-shot learning), and confirm this observation on our newly introduced UnRel dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑