arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.14969cs.CV
收藏

MMNet:用于指称图像分割的多掩码网络

MMNet: Multi-Mask Network for Referring Image Segmentation

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology Beijing(北京科技大学);University of Chinese Academy of Sciences(中国科学院大学)

Yichen Yan, Xingjian He, Wenxuan Wan, Jing Liu

AI总结:

本文提出端到端多掩码网络MMNet,通过注意力机制生成多个查询及对应分割掩码并加权融合,有效降低语言表达随机性,在RefCOCO、RefCOCO+和G-Ref数据集上无需后处理即取得优于现有方法的性能。

摘要:

指称图像分割旨在根据自然语言表达从图像中分割出所指的对象。然而,由于文本与图像之间不同的数据特性,以及多样对象和无约束语言表达引入的随机性,该任务具有挑战性。以往大多数工作侧重于改进跨模态特征融合,但未能充分解决由多样对象和无约束语言引起的固有不确定性。为解决这些问题,我们提出了一种端到端的用于指称图像分割的多掩码网络(MMNet)。我们首先结合图像和语言,然后采用注意力机制生成多个查询,这些查询代表语言表达的不同方面。随后,我们利用这些查询生成一系列相应的分割掩码,并为每个掩码分配一个反映其重要性的分数。最终结果通过所有掩码的加权求和获得,这大大降低了语言表达的随机性。与最先进的方法相比,我们提出的框架在两个最常用的数据集 RefCOCO、RefCOCO+ 和 G-Ref 上表现出优越的性能,且无需任何后处理。这进一步验证了我们所提出框架的有效性。

英文摘要:

Referring image segmentation aims to segment an object referred to by natural language expression from an image. However, this task is challenging due to the distinct data properties between text and image, and the randomness introduced by diverse objects and unrestricted language expression. Most of previous work focus on improving cross-modal feature fusion while not fully addressing the inherent uncertainty caused by diverse objects and unrestricted language. To tackle these problems, we propose an end-to-end Multi-Mask Network for referring image segmentation(MMNet). we first combine picture and language and then employ an attention mechanism to generate multiple queries that represent different aspects of the language expression. We then utilize these queries to produce a series of corresponding segmentation masks, assigning a score to each mask that reflects its importance. The final result is obtained through the weighted sum of all masks, which greatly reduces the randomness of the language expression. Our proposed framework demonstrates superior performance compared to state-of-the-art approaches on the two most commonly used datasets, RefCOCO, RefCOCO+ and G-Ref, without the need for any post-processing. This further validates the efficacy of our proposed framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑