arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.08230cs.CVcs.LG
收藏

场景图生成中新颖合成的图密度感知损失

Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation

University of Guelph(圭尔夫大学);Vector Institute for Artificial Intelligence(人工智能向量研究所);Element AI(Element AI 公司);University of Cambridge(剑桥大学);Mila, Université de Montréal(蒙特利尔大学 Mila 研究院);CIFAR(加拿大高等研究院)

Boris Knyazev, Harm de Vries, Cătălina Cangea, Graham W. Taylor, Aaron Courville, Eugene Belilovsky

摘要:

场景图生成(SGG)的目的是预测输入图像的图形结构的描述,在对象和它们之间的关系的形式。这项任务对于视觉和语言界面的进步越来越有用。在这里,重要的是-但具有挑战性的-表现在新的(zero-shot)或罕见的(Few-Shot)对象和关系的组成。在本文中,我们确定了两个关键问题,限制这种推广。首先,我们表明,在这项任务中使用的标准损失是无意中场景图密度的函数。这导致在训练过程中忽略大型稀疏图中的单个边缘,即使这些包含对泛化很重要的各种Few-Shot示例。其次,关系的频率可以在该任务中产生强烈的偏差,使得预测最频繁关系的盲模型实现良好的性能。因此,一些最先进的模型利用这种偏差来改善结果。我们发现,这些模型在推广到稀有成分的能力方面受到的影响最大,在Visual Genome数据集及其更新的改进版本GQA上评估了两种不同的模型。为了解决这些问题,我们引入了一个密度归一化的边缘损失,它提供了超过两倍的改进,在某些泛化指标。相比其他作品在这个方向上,我们的增强只需要几行代码,没有增加计算成本。我们还强调了使用现有指标准确评估模型的困难,特别是在零/少数镜头上,并引入了一种新的加权指标。

英文摘要:

Scene graph generation (SGG) aims to predict graph-structured descriptions of input images, in the form of objects and relationships between them. This task is becoming increasingly useful for progress at the interface of vision and language. Here, it is important - yet challenging - to perform well on novel (zero-shot) or rare (few-shot) compositions of objects and relationships. In this paper, we identify two key issues that limit such generalization. Firstly, we show that the standard loss used in this task is unintentionally a function of scene graph density. This leads to the neglect of individual edges in large sparse graphs during training, even though these contain diverse few-shot examples that are important for generalization. Secondly, the frequency of relationships can create a strong bias in this task, such that a blind model predicting the most frequent relationship achieves good performance. Consequently, some state-of-the-art models exploit this bias to improve results. We show that such models can suffer the most in their ability to generalize to rare compositions, evaluating two different models on the Visual Genome dataset and its more recent, improved version, GQA. To address these issues, we introduce a density-normalized edge loss, which provides more than a two-fold improvement in certain generalization metrics. Compared to other works in this direction, our enhancements require only a few lines of code and no added computational cost. We also highlight the difficulty of accurately evaluating models using existing metrics, especially on zero/few shots, and introduce a novel weighted metric.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑