arXiv 2408.147762024-11-27cs.CVcs.AI

MROVSeg：打破视觉语言模型在开放词汇图像分割中的分辨率诅咒

MROVSeg: Breaking the Resolution Curse of Vision-Language Models in Open-Vocabulary Image Segmentation

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）

Yuanbing Zhu, Bingke Zhu, Yingying Chen, Yunfang Niu, Ming Tang, Jinqiao Wang

备注：: Technical report

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对开放词汇图像分割中视觉语言模型（如CLIP）因低分辨率预训练导致的细节丢失问题，提出MROVSeg多分辨率训练框架，通过滑动窗口和多分辨率适配器等组件，在单CLIP骨干上实现高分辨率分割，提升性能并降低计算开销。

摘要：

预训练视觉语言模型（VLMs），例如CLIP，正越来越多地用于弥合开放词汇图像分割中开放词汇与闭词汇识别之间的差距。由于VLMs通常使用低分辨率图像（例如224×224）进行预训练，大多数先前方法仅在降采样图像上操作。我们质疑这种设计，因为低分辨率特征往往无法保留精细细节。典型解决方案是为高分辨率输入采用额外图像骨干网络，但会引入显著计算开销。因此，我们提出MROVSeg，一种基于单个预训练CLIP骨干网络的开放词汇图像分割多分辨率训练框架，它使用滑动窗口将高分辨率输入分割为均匀补丁，每个补丁匹配训练良好的图像编码器的输入尺寸。其关键组件包括多分辨率适配器（Multi-Res Adapter），通过与多分辨率特征交互恢复空间几何结构并掌握补丁间的局部-全局对应关系。为实现准确分割，我们引入多粒度掩码注意力（Multi-grained Masked Attention）方案，从多分辨率CLIP特征向对象查询聚合多粒度语义。通过全面实验，我们证明MROVSeg在成熟的开放词汇图像分割基准上的优越性，为开放词汇图像分割建立新标准。

英文摘要：

Pretrained vision-language models (VLMs), \eg CLIP, are increasingly used to bridge the gap between open- and close-vocabulary recognition in open-vocabulary image segmentation. As VLMs are generally pretrained with low-resolution images (e.g. $224\times224$), most previous methods operate only on downscaled images. We question this design as low resolution features often fail to preserve fine details. A typical solution is to employ additional image backbones for high-resolution inputs, but it also introduce significant computation overhead. Therefore, we propose MROVSeg, a multi-resolution training framework for open-vocabulary image segmentation with a single pretrained CLIP backbone, that uses sliding windows to slice the high-resolution input into uniform patches, each matching the input size of the well-trained image encoder. Its key components include a Multi-Res Adapter, which restores the spatial geometry and grasps local-global correspondences across patches by interacting with multi-resolution features. To achieve accurate segmentation, we introduce Multi-grained Masked Attention scheme to aggregate multi-grained semantics from multi-resolution CLIP features to object queries. Through comprehensive experiments, we demonstrate the superiority of MROVSeg on well-established open-vocabulary image segmentation benchmarks, establishing new standards for open-vocabulary image segmentation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。