arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.02598cs.CV
收藏

用对抗网络生成三元组构建场景图

Generating Triples with Adversarial Networks for Scene Graph Construction

Rensselaer Polytechnic Institute(伦斯勒理工学院);Air Force Research Laboratory(美国空军研究实验室)

Matthew Klawonn, Eric Heim

摘要:

在深度学习成功的推动下,计算机视觉研究已经开始从目标检测和图像分类转向更复杂的任务,如图像描述或视觉问答。激发这种努力的是模型不仅要捕捉图像中存在的对象,而且要捕捉场景的更细粒度的方面,例如对象及其属性之间的关系。场景图提供了一种用于捕获图像的这些方面的正式构造。尽管如此,最近只有少数努力从图像生成场景图。以前的作品限制自己的设置限界框信息是在训练时间,并不试图生成场景图的属性。在本文中,我们提出了一种方法,生成对抗网络的最新进展的基础上,克服这些不足。我们采取的方法是首先生成小的子图,每个子图描述一个单一的语句,从一个特定的区域的输入图像选择使用注意力机制的场景。通过这样做,我们的方法是能够产生部分的场景图的属性信息,而不需要限界框标签。然后,由这些子图构造出完整的场景图。我们表明,我们的模型改进了以前的工作,在场景图生成的最先进的数据集和公认的指标。此外,我们证明了我们的模型能够处理更大的词汇量比以前的工作已经尝试。

英文摘要:

Driven by successes in deep learning, computer vision research has begun to move beyond object detection and image classification to more sophisticated tasks like image captioning or visual question answering. Motivating such endeavors is the desire for models to capture not only objects present in an image, but more fine-grained aspects of a scene such as relationships between objects and their attributes. Scene graphs provide a formal construct for capturing these aspects of an image. Despite this, there have been only a few recent efforts to generate scene graphs from imagery. Previous works limit themselves to settings where bounding box information is available at train time and do not attempt to generate scene graphs with attributes. In this paper we propose a method, based on recent advancements in Generative Adversarial Networks, to overcome these deficiencies. We take the approach of first generating small subgraphs, each describing a single statement about a scene from a specific region of the input image chosen using an attention mechanism. By doing so, our method is able to produce portions of the scene graphs with attribute information without the need for bounding box labels. Then, the complete scene graph is constructed from these subgraphs. We show that our model improves upon prior work in scene graph generation on state-of-the-art data sets and accepted metrics. Further, we demonstrate that our model is capable of handling a larger vocabulary size than prior work has attempted.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑