基于嵌入调制的开放词汇全景分割
Open-vocabulary Panoptic Segmentation with Embedding Modulation
AI总结：
提出 OPSNet 框架，通过嵌入调制模块连接分割模型与 CLIP，以较少额外数据实现开放与闭集词汇全景分割的先进性能。
摘要：
开放词汇图像分割因其在现实世界中的关键应用而受到越来越多的关注。传统的闭集词汇分割方法无法刻画新颖物体，而近期若干开放词汇尝试的结果并不理想，即在闭集词汇上性能显著下降，并且对额外数据有大量需求。为此，我们提出 OPSNet，这是一个用于开放词汇全景分割的全能且数据高效的框架。具体而言，精心设计的 Embedding Modulation（嵌入调制）模块连同若干细致的组件，使分割模型与视觉-语言良好对齐的 CLIP 编码器之间能够进行充分的嵌入增强和信息交换，从而在开放词汇和闭集词汇设置下均取得卓越的分割性能，并且对额外数据的需求少得多。我们在多种情况下跨多个数据集（例如 COCO、ADE20K、Cityscapes 和 PascalContext）进行了广泛的实验评估，所提出的 OPSNet 取得了最先进的结果，这证明了所提方法的有效性和通用性。代码和训练好的模型将公开发布。
英文摘要：
Open-vocabulary image segmentation is attracting increasing attention due to its critical applications in the real world. Traditional closed-vocabulary segmentation methods are not able to characterize novel objects, whereas several recent open-vocabulary attempts obtain unsatisfactory results, i.e., notable performance reduction on the closed vocabulary and massive demand for extra data. To this end, we propose OPSNet, an omnipotent and data-efficient framework for Open-vocabulary Panoptic Segmentation. Specifically, the exquisitely designed Embedding Modulation module, together with several meticulous components, enables adequate embedding enhancement and information exchange between the segmentation model and the visual-linguistic well-aligned CLIP encoder, resulting in superior segmentation performance under both open- and closed-vocabulary settings with much fewer need of additional data. Extensive experimental evaluations are conducted across multiple datasets (e.g., COCO, ADE20K, Cityscapes, and PascalContext) under various circumstances, where the proposed OPSNet achieves state-of-the-art results, which demonstrates the effectiveness and generality of the proposed approach. The code and trained models will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
