arXiv 2410.080212024-10-25cs.CV

OneRef：采用掩码指称建模的统一单塔表达式定位与分割

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））

Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, Changsheng Xu

备注：: Accepted by NeurIPS 2024. The project page: https://github.com/linhuixiao/OneRef

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出 OneRef 单塔统一视觉语言框架，并通过 MRefM 掩码指称建模学习跨模态指称关系，在定位与分割任务上均达到 SoTA。

摘要：

受视觉和语言分离编码的限制，现有的定位和指称分割工作严重依赖笨重的基于 Transformer 的融合编码器/解码器以及多种早期交互技术。同时，当前的掩码视觉语言建模（MVLM）无法捕捉指称任务中图像与文本之间细微的指称关系。本文提出 OneRef，这是一个构建于模态共享单塔 transformer 之上的极简指称框架，统一了视觉和语言特征空间。为建模指称关系，作者引入一种名为 Mask Referring Modeling（MRefM，掩码指称建模）的新型 MVLM 范式，它同时包含指称感知掩码图像建模和指称感知掩码语言建模。两个模块不仅重建与模态相关的内容，还重建跨模态指称内容。在 MRefM 中，作者提出一种指称感知动态图像掩码策略，该策略能够感知被指称区域，而不是依赖固定比例或通用随机掩码方案。借助统一的视觉语言特征空间并结合 MRefM 建模指称关系的能力，该方法无需借助各种复杂技术即可直接回归指称结果。本文方法持续超越现有方法，在定位和分割任务上均取得 SoTA 性能，为未来研究提供了有价值的见解。代码和模型可在 https://github.com/linhuixiao/OneRef 获取。

英文摘要：

Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。