arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2110.04435cs.CVcs.CL
收藏

基于两级视觉线索增强网络的参考图像分割

Two-stage Visual Cues Enhancement Network for Referring Image Segmentation

Fudan University(复旦大学);Meituan(美团)

Yang Jiao, Zequn Jie, Weixin Luo, Jingjing Chen, Yu-Gang Jiang, Xiaolin Wei, Lin Ma

摘要:

参照图像分割(RIS)的目的是从给定的自然语言表达式所参照的图像中分割出目标对象。图像中表达方式的多样性和灵活性以及视觉内容的复杂性对RIS模型研究表达方式中的词语与图像中呈现的对象之间的细粒度匹配行为提出了更高的要求。然而,当所指对象的视觉线索不足时,这种匹配行为很难被学习和捕获,因为具有弱视觉线索的所指对象往往容易被边界处的杂乱背景混淆,甚至被图像中的显著对象淹没。而视觉线索不足的问题不能像以前的工作那样通过跨模态融合机制来处理。在本文中,我们解决这个问题,从一个新的角度来提高视觉信息的参考设计一个两阶段的视觉线索增强网络(TV-Net),其中一个新的检索和丰富计划(RES)和自适应多分辨率特征融合(AMF)模块提出。通过两阶段的增强,我们提出的TV-Net在学习自然语言表达和图像之间的细粒度匹配行为方面具有更好的性能,特别是当所指对象的视觉信息不足时,从而产生更好的分割结果。进行了大量的实验来验证所提出的方法在RIS任务上的有效性,我们提出的TV-Net在四个基准数据集上超过了最先进的方法。

英文摘要:

Referring Image Segmentation (RIS) aims at segmenting the target object from an image referred by one given natural language expression. The diverse and flexible expressions as well as complex visual contents in the images raise the RIS model with higher demands for investigating fine-grained matching behaviors between words in expressions and objects presented in images. However, such matching behaviors are hard to be learned and captured when the visual cues of referents (i.e. referred objects) are insufficient, as the referents with weak visual cues tend to be easily confused by cluttered background at boundary or even overwhelmed by salient objects in the image. And the insufficient visual cues issue can not be handled by the cross-modal fusion mechanisms as done in previous work. In this paper, we tackle this problem from a novel perspective of enhancing the visual information for the referents by devising a Two-stage Visual cues enhancement Network (TV-Net), where a novel Retrieval and Enrichment Scheme (RES) and an Adaptive Multi-resolution feature Fusion (AMF) module are proposed. Through the two-stage enhancement, our proposed TV-Net enjoys better performances in learning fine-grained matching behaviors between the natural language expression and image, especially when the visual information of the referent is inadequate, thus produces better segmentation results. Extensive experiments are conducted to validate the effectiveness of the proposed method on the RIS task, with our proposed TV-Net surpassing the state-of-the-art approaches on four benchmark datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑