迈向跨全层级视觉目标粒度的统一指代表达分割
Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities
AI总结：
本文提出多粒度指代表达分割任务MRES及RefCOCOm基准，构建最大数据集MRES-32M，并设计统一多模态大模型UniRES++，在多个基准上实现最先进性能。
摘要：
指代表达分割（RES）旨在分割与描述性语言表达相匹配的实体掩码。传统的RES方法主要处理对象级别的定位，而现实场景需要一个更通用的框架来处理多层级目标粒度，如多对象、单对象或部件级别的指代。由于用户描述目标的方式多样且细微，这带来了巨大挑战。然而，现有数据集和模型主要聚焦于为对象级目标定位设计定位专家，缺乏更实用的多粒度RES所需的数据资源和统一框架。本文进一步推进视觉粒度统一RES任务。为克服数据稀缺限制，我们引入了新的多粒度指代表达分割（MRES）任务，并发布了RefCOCOm基准，包含部件级标注以推进更细粒度的视觉理解。此外，我们构建了MRES-32M，这是最大的视觉定位数据集，包含超过32.2M掩码和配对描述，覆盖1M图像，专为部件级视觉-语言定位设计。为应对多粒度RES挑战，我们提出UniRES++，一个统一的多模态大语言模型，集成对象级和部件级RES任务。UniRES++针对细粒度视觉特征探索进行了专门设计。凭借联合模型架构和参数，UniRES++在多个基准上取得最先进性能，包括用于MRES的RefCOCOm、用于泛化RES的gRefCOCO，以及用于经典RES的RefCOCO、RefCOCO+和RefCOCOg。为促进未来多粒度视觉定位研究，RefCOCOm基准、MRES-32M数据集和UniRES++模型将公开于https://github.com/Rubics-Xuan/MRES。
英文摘要：
Referring expression segmentation (RES) aims at segmenting the entities' masks that match the descriptive language expression. While traditional RES methods primarily address object-level grounding, real-world scenarios demand a more versatile framework that can handle multiple levels of target granularity, such as multi-object, single object or part-level references. This introduces great challenges due to the diverse and nuanced ways users describe targets. However, existing datasets and models mainly focus on designing grounding specialists for object-level target localization, lacking the necessary data resources and unified frameworks for the more practical multi-grained RES. In this paper, we take a step further towards visual granularity unified RES task. To overcome the limitation of data scarcity, we introduce a new multi-granularity referring expression segmentation (MRES) task, alongside the RefCOCOm benchmark, which includes part-level annotations for advancing finer-grained visual understanding. In addition, we create MRES-32M, the largest visual grounding dataset, comprising over 32.2M masks and captions across 1M images, specifically designed for part-level vision-language grounding. To tackle the challenges of multi-granularity RES, we propose UniRES++, a unified multimodal large language model that integrates object-level and part-level RES tasks. UniRES++ incorporates targeted designs for fine-grained visual feature exploration. With the joint model architecture and parameters, UniRES++ achieves state-of-the-art performance across multiple benchmarks, including RefCOCOm for MRES, gRefCOCO for generalized RES, and RefCOCO, RefCOCO+, RefCOCOg for classic RES. To foster future research into multi-grained visual grounding, our RefCOCOm benchmark, MRES-32M dataset and model UniRES++ will be publicly available at https://github.com/Rubics-Xuan/MRES.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
