arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.04863cs.CVcs.LG
收藏

基于场景和知识图谱的符号图像检测

Symbolic image detection using scene and knowledge graphs

University of Pittsburgh(匹兹堡大学)

Nasrin Kalanat, Adriana Kovashka

摘要:

有时,图像传达的意义超出了它们所包含的对象列表;相反,图像可能表达一个强大的信息来影响观众的思想。推断此消息需要对对象之间的关系进行推理,以及对组件的一般常识性知识。在本文中,我们使用场景图(图像的图形表示)来捕获视觉组件。此外,我们使用从ConceptNet中提取的事实生成知识图谱来推理对象和属性。为了检测符号,我们提出了一个神经网络框架命名为SKG-Sym。该框架首先使用图卷积网络生成图像的场景图及其知识图谱的表示。然后,该框架融合表示,并使用MLP对它们进行分类。我们进一步扩展了网络,使用一个注意力机制来学习图形表示的重要性。我们在广告数据集上评估了我们的方法,并将其与基线符号分类方法(ResNet和VGG)进行了比较。结果表明,我们的方法在F-score方面优于ResNet,基于注意力的机制与VGG竞争,同时具有更低的模型复杂度。

英文摘要:

Sometimes the meaning conveyed by images goes beyond the list of objects they contain; instead, images may express a powerful message to affect the viewers' minds. Inferring this message requires reasoning about the relationships between the objects, and general common-sense knowledge about the components. In this paper, we use a scene graph, a graph representation of an image, to capture visual components. In addition, we generate a knowledge graph using facts extracted from ConceptNet to reason about objects and attributes. To detect the symbols, we propose a neural network framework named SKG-Sym. The framework first generates the representations of the scene graph of the image and its knowledge graph using Graph Convolution Network. The framework then fuses the representations and uses an MLP to classify them. We extend the network further to use an attention mechanism which learn the importance of the graph representations. We evaluate our methods on a dataset of advertisements, and compare it with baseline symbolism classification methods (ResNet and VGG). Results show that our methods outperform ResNet in terms of F-score and the attention-based mechanism is competitive with VGG while it has much lower model complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑