arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.00515cs.CVcs.CL
收藏

基于语言结构指导的上下文建模的参考图像分割

Linguistic Structure Guided Context Modeling for Referring Image Segmentation

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Sun Yat-sen University(中山大学);Tencent Marketing Solution(腾讯营销解决方案)

Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, Jizhong Han

摘要:

指称图像分割的目的是预测由自然语言句子指称的对象的前景掩模。句子的多模态语境是区分指称对象和背景的关键。现有的方法要么不充分或冗余模型的多模态上下文。为了解决这个问题,我们提出了一个“传播-传播-分发”计划,通过跨模态交互建模多模态上下文,并实现该计划作为一个新的语言结构引导上下文建模(LSCM)模块。我们的LSCM模块建立了一个依存句法分析树抑制词图(DPT-WG),它引导所有的词包括有效的多模态上下文的句子,同时通过三个步骤排除干扰的多模态特征,即,聚集、约束传播和分发。在四个基准上的大量实验表明,我们的方法优于所有以前的国家的最先进的。

英文摘要:

Referring image segmentation aims to predict the foreground mask of the object referred by a natural language sentence. Multimodal context of the sentence is crucial to distinguish the referent from the background. Existing methods either insufficiently or redundantly model the multimodal context. To tackle this problem, we propose a "gather-propagate-distribute" scheme to model multimodal context by cross-modal interaction and implement this scheme as a novel Linguistic Structure guided Context Modeling (LSCM) module. Our LSCM module builds a Dependency Parsing Tree suppressed Word Graph (DPT-WG) which guides all the words to include valid multimodal context of the sentence while excluding disturbing ones through three steps over the multimodal feature, i.e., gathering, constrained propagation and distributing. Extensive experiments on four benchmarks demonstrate that our method outperforms all the previous state-of-the-arts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑