arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.08007cs.CV
收藏

揭示物体之外的部分:迈向更细粒度的指代表达式分割

Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentation

Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院);University of Science and Technology Beijing (USTB)(北京科技大学)

Wenxuan Wang, Tongtian Yue, Yisi Zhang, Longteng Guo, Xingjian He, Xinlong Wang, Jing Liu

AI总结:

本文提出更细粒度的多粒度指代表达式分割(MRES)任务,构建了包含超32.2M掩码的MRES-32M数据集与RefCOCOm基准,并设计统一物体级与部分级定位的模型UniRES,实验证明其优于现有方法。

摘要:

指代表达式分割(RES)旨在分割与描述性自然语言表达式匹配的实体前景掩码。以往的经典RES任务数据集和方法严重依赖于一个表达式必须指代物体级目标的先验假设。本文进一步探索更细粒度的部分级RES任务。为推动物体级RES任务向更细粒度的视觉-语言理解发展,我们提出了新的多粒度指代表达式分割(MRES)任务,并通过人工标注构建了名为RefCOCOm的评估基准。借助我们自动化的模型辅助数据引擎,我们构建了目前最大的视觉定位数据集MRES-32M,包含在1M图像上生成的超过32.2M高质量掩码和描述。此外,设计了一个简单而强大的模型UniRES,以完成统一的物体级和部分级定位任务。在RefCOCOm上的MRES任务和三个经典RES任务数据集(即RefCOCO(+/g))上的大量实验表明,我们的方法优于以往的最先进方法。为促进未来细粒度视觉定位研究,我们的基准RefCOCOm、MRES-32M数据集和UniRES模型将公开于https://github.com/Rubics-Xuan/MRES。

英文摘要:

Referring expression segmentation (RES) aims at segmenting the foreground masks of the entities that match the descriptive natural language expression. Previous datasets and methods for classic RES task heavily rely on the prior assumption that one expression must refer to object-level targets. In this paper, we take a step further to finer-grained part-level RES task. To promote the object-level RES task towards finer-grained vision-language understanding, we put forward a new multi-granularity referring expression segmentation (MRES) task and construct an evaluation benchmark called RefCOCOm by manual annotations. By employing our automatic model-assisted data engine, we build the largest visual grounding dataset namely MRES-32M, which comprises over 32.2M high-quality masks and captions on the provided 1M images. Besides, a simple yet strong model named UniRES is designed to accomplish the unified object-level and part-level grounding task. Extensive experiments on our RefCOCOm for MRES and three datasets (i.e., RefCOCO(+/g) for classic RES task demonstrate the superiority of our method over previous state-of-the-art methods. To foster future research into fine-grained visual grounding, our benchmark RefCOCOm, the MRES-32M dataset and model UniRES will be publicly available at https://github.com/Rubics-Xuan/MRES

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑