arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.04053cs.CVcs.MM
收藏

弱指称表达式接地的判别式三元组匹配与重构

Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding

University of Liverpool(利物浦大学);Xi’an Jiaotong-Liverpool University(西交利物浦大学);Beihang University(北京航空航天大学)

Mingjie Sun, Jimin Xiao, Eng Gee Lim, Si Liu, John Y. Goulermas

摘要:

在本文中,我们正在解决弱监督的指涉表达式接地任务,根据查询语句在图像中定位所指对象,其中图像区域和查询之间的映射在训练阶段不可用。在传统的方法中,首先选择一个与指称表达式最匹配的对象区域,然后从该区域重构出查询语句,重构差作为反向传播的损失。现有的方法,然而,进行匹配和重建近似,因为他们忽略了这一事实,即匹配的正确性是未知的。为了克服这一限制,一个判别三元组设计在这里作为解决方案的基础,通过它可以转换成一个或多个判别三元组在一个非常可扩展的方式查询。基于判别三元组,我们进一步提出了三元组级匹配和重建模块,这些模块对于弱监督训练来说是轻量级的,但有效的,使其比以前的最先进的方法轻三倍,速度快三倍。我们的工作的一个重要优点是其优越的性能,尽管简单而整洁的设计。具体而言,当在RefCOCO(39.21%)、RefCOCO+(39.18%)和RefCOCOg(43.24%)数据集上进行评估时,所提出的方法实现了新的最先进的准确度,分别比之前的方法高出4.17%、4.08%和7.8%。

英文摘要:

In this paper, we are tackling the weakly-supervised referring expression grounding task, for the localization of a referent object in an image according to a query sentence, where the mapping between image regions and queries are not available during the training stage. In traditional methods, an object region that best matches the referring expression is picked out, and then the query sentence is reconstructed from the selected region, where the reconstruction difference serves as the loss for back-propagation. The existing methods, however, conduct both the matching and the reconstruction approximately as they ignore the fact that the matching correctness is unknown. To overcome this limitation, a discriminative triad is designed here as the basis to the solution, through which a query can be converted into one or multiple discriminative triads in a very scalable way. Based on the discriminative triad, we further propose the triad-level matching and reconstruction modules which are lightweight yet effective for the weakly-supervised training, making it three times lighter and faster than the previous state-of-the-art methods. One important merit of our work is its superior performance despite the simple and neat design. Specifically, the proposed method achieves a new state-of-the-art accuracy when evaluated on RefCOCO (39.21%), RefCOCO+ (39.18%) and RefCOCOg (43.24%) datasets, that is 4.17%, 4.08% and 7.8% higher than the previous one, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑