arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1608.00187cs.CV
收藏

基于语言先验的视觉关系检测

Visual Relationship Detection with Language Priors

Stanford University(斯坦福大学)

Cewu Lu, Ranjay Krishna, Michael Bernstein, Li Fei-Fei

摘要:

视觉关系捕捉图像中成对对象之间的各种各样的交互(例如,“人骑自行车”和“人推自行车”)。因此,可能的关系的集合非常大,并且难以为所有可能的关系获得足够的训练样本。由于这种限制,以前的工作视觉关系检测集中在预测只有少数的关系。虽然大多数关系并不频繁,但它们的宾语(例如“man”和“bicycle”)和谓语(例如“riding”和“pushing”)独立出现的频率更高。我们提出了一个模型,使用这种洞察力,训练视觉模型的对象和谓词单独,后来将它们结合在一起,预测多个关系的图像。我们通过利用语义词嵌入的语言先验来微调预测关系的可能性,从而改进了先前的工作。我们的模型可以扩展到从几个例子中预测数千种类型的关系。此外,我们将预测关系中的对象定位为图像中的边界框。我们进一步证明,理解关系可以提高基于内容的图像检索。

英文摘要:

Visual relationships capture a wide variety of interactions between pairs of objects in images (e.g. "man riding bicycle" and "man pushing bicycle"). Consequently, the set of possible relationships is extremely large and it is difficult to obtain sufficient training examples for all possible relationships. Because of this limitation, previous work on visual relationship detection has concentrated on predicting only a handful of relationships. Though most relationships are infrequent, their objects (e.g. "man" and "bicycle") and predicates (e.g. "riding" and "pushing") independently occur more frequently. We propose a model that uses this insight to train visual models for objects and predicates individually and later combines them together to predict multiple relationships per image. We improve on prior work by leveraging language priors from semantic word embeddings to finetune the likelihood of a predicted relationship. Our model can scale to predict thousands of types of relationships from a few examples. Additionally, we localize the objects in the predicted relationships as bounding boxes in the image. We further demonstrate that understanding relationships can improve content based image retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑