arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2003.12058cs.CV
收藏

接地情况识别

Grounded Situation Recognition

Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)

Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, Aniruddha Kembhavi

摘要:

我们介绍接地情况识别(GSR),一个任务,需要产生结构化的语义摘要的图像描述:主要活动,从事活动的实体与他们的角色(如代理,工具),和边界框接地的实体。GSR提出了重要的技术挑战:识别语义显着性,分类和本地化的一个大的和不同的实体集,克服语义稀疏,和消除歧义的角色。此外,与字幕不同,GSR的评估很简单。为了研究这个新任务,我们创建了Situations With Groundings(SWiG)数据集,它将278,336个边界框接地添加到imsitu数据集中的11,538个实体类中。我们提出了一个联合情况定位器,并发现联合预测情况和接地与端到端的培训轻松地优于整个接地度量套件的独立培训,相对增益在8%和32%之间。最后,我们展示了我们的模型所支持的三个令人兴奋的未来方向的初步研究结果:条件查询,视觉链接和接地语义感知图像检索。代码和数据可在https://prior.allenai.org/projects/gsr获得。

英文摘要:

We introduce Grounded Situation Recognition (GSR), a task that requires producing structured semantic summaries of images describing: the primary activity, entities engaged in the activity with their roles (e.g. agent, tool), and bounding-box groundings of entities. GSR presents important technical challenges: identifying semantic saliency, categorizing and localizing a large and diverse set of entities, overcoming semantic sparsity, and disambiguating roles. Moreover, unlike in captioning, GSR is straightforward to evaluate. To study this new task we create the Situations With Groundings (SWiG) dataset which adds 278,336 bounding-box groundings to the 11,538 entity classes in the imsitu dataset. We propose a Joint Situation Localizer and find that jointly predicting situations and groundings with end-to-end training handily outperforms independent training on the entire grounding metric suite with relative gains between 8% and 32%. Finally, we show initial findings on three exciting future directions enabled by our models: conditional querying, visual chaining, and grounded semantic aware image retrieval. Code and data available at https://prior.allenai.org/projects/gsr.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑