arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2008.06254cs.CV
收藏

ConsNet:学习一致性图的Zero-Shot人机交互检测

ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection

Wuhan University(武汉大学);State University of New York at Buffalo(纽约州立大学布法罗分校);Peng Cheng Laboratory(鹏城实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Ye Liu, Junsong Yuan, Chang Wen Chen

摘要:

我们考虑的问题,人-物交互(HOI)检测,其目的是定位和识别HOI实例的形式的<人,动作,对象>的图像。现有的研究大多将HOI视为个体交互范畴,无法处理动作标签的长尾分布和多义性问题。我们认为,多层次的对象,动作和交互之间的语义表征生成罕见的或以前看不见的HOI的强有力的线索。利用HOI标签的组成和关系特性,我们提出了ConsNet,一个知识感知的框架,明确编码对象,动作和交互之间的关系到一个无向图称为一致性图,并利用图注意力网络(GATs)传播知识之间的HOI类别以及它们的成分。该模型以候选人-物对的视觉特征和HOI标签的词嵌入作为输入,将其映射到视觉-语义联合嵌入空间,通过度量其相似性得到检测结果。我们在具有挑战性的V-COCO和HICO-DET数据集上对我们的模型进行了广泛的评估,结果验证了我们的方法在全监督和zero-shot设置下都优于最先进的方法。代码可在https://github.com/yeliudev/ConsNet上获得。

英文摘要:

We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of <human, action, object> in images. Most existing works treat HOIs as individual interaction categories, thus can not handle the problem of long-tail distribution and polysemy of action labels. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Leveraging the compositional and relational peculiarities of HOI labels, we propose ConsNet, a knowledge-aware framework that explicitly encodes the relations among objects, actions and interactions into an undirected graph called consistency graph, and exploits Graph Attention Networks (GATs) to propagate knowledge among HOI categories as well as their constituents. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities. We extensively evaluate our model on the challenging V-COCO and HICO-DET datasets, and results validate that our approach outperforms state-of-the-arts under both fully-supervised and zero-shot settings. Code is available at https://github.com/yeliudev/ConsNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑