arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.09554cs.CV
收藏

迈向鲁棒的指代表达图像分割

Towards Robust Referring Image Segmentation

Peking University（北京大学）；ETH Zurich（苏黎世联邦理工学院）；The University of Sydney（悉尼大学）

Jianzong Wu, Xiangtai Li, Xia Li, Henghui Ding, Yunhai Tong, Dacheng Tao

备注：
update more results
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对文本描述错误或目标不存在的问题，本文提出R-RIS任务、三个数据集、统一指标及RefSegformer模型，并在RIS与R-RIS上取得SOTA基线。

摘要：

指代表达图像分割（Referring Image Segmentation，RIS）是一项基础的视觉-语言任务，旨在根据文本描述输出目标对象掩码。许多研究已在RIS上取得显著进展，包括不同的融合方法设计。在本文中，我们探讨一个关键问题：“如果文本描述错误或具有误导性会怎样？”例如，所描述的对象并不存在于图像中。我们将此类句子称为否定句。然而，现有RIS解决方案无法处理这种情形。为此，我们提出一种新的RIS形式，命名为鲁棒指代表达图像分割（Robust Referring Image Segmentation，R-RIS）。除常规的肯定文本输入外，它还考虑否定句输入。为推动这一新任务，我们通过为现有RIS数据集增加否定句，构建了三个R-RIS数据集，并提出新指标，以统一方式评估两类输入。此外，我们提出一种基于transformer的新模型RefSegformer，其具有基于token的视觉与语言融合模块。通过添加额外的空白token，我们的设计可轻松扩展到R-RIS设定。所提出的RefSegformer在RIS和R-RIS数据集上均取得最先进结果，为两种设定建立了坚实基线。项目页面位于https://github.com/jianzongwu/robust-ref-seg。

英文摘要：

Referring Image Segmentation (RIS) is a fundamental vision-language task that outputs object masks based on text descriptions. Many works have achieved considerable progress for RIS, including different fusion method designs. In this work, we explore an essential question, ``What if the text description is wrong or misleading?'' For example, the described objects are not in the image. We term such a sentence as a negative sentence. However, existing solutions for RIS cannot handle such a setting. To this end, we propose a new formulation of RIS, named Robust Referring Image Segmentation (R-RIS). It considers the negative sentence inputs besides the regular positive text inputs. To facilitate this new task, we create three R-RIS datasets by augmenting existing RIS datasets with negative sentences and propose new metrics to evaluate both types of inputs in a unified manner. Furthermore, we propose a new transformer-based model, called RefSegformer, with a token-based vision and language fusion module. Our design can be easily extended to our R-RIS setting by adding extra blank tokens. Our proposed RefSegformer achieves state-of-the-art results on both RIS and R-RIS datasets, establishing a solid baseline for both settings. Our project page is at \url{https://github.com/jianzongwu/robust-ref-seg}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑