揭示原型知识用于弱监督开放词汇语义分割
Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation
AI总结：
本文针对弱监督开放词汇语义分割中组token训练与推理粒度不一致问题，提出不可学习原型正则化NPR和原型引导分割网络PGSeg，通过图像与文本多层级原型监督增强分割性能，并在多个基准上取得最优结果。
摘要：
本文研究弱监督开放词汇语义分割（WOVSS）问题，即仅利用图像-文本对来学习分割任意类别的对象。现有工作通过引入显式分组识别来增强普通视觉Transformer，即使用若干组token/质心对图像token进行聚类并执行组-文本对齐。然而，这些方法在组token的使用上存在粒度不一致问题：在训练和推理阶段分别以全对一和一对一的方式进行对齐。我们认为这种差异源于缺乏对每个组token的精细监督。为弥合这一粒度差距，本文从原型知识出发探索对组token的显式监督。为此，本文提出不可学习原型正则化（NPR），其中从源特征估计不可学习原型以充当监督，并实现组token的对比匹配。该正则化促使组token以更少冗余分割对象并捕获更全面的语义区域，从而提高紧凑性和丰富性。基于NPR，我们提出原型引导分割网络（PGSeg），通过利用来自图像和文本在不同层级上的原型源来引入多模态正则化，逐步以多样化的原型模式增强分割能力。实验结果表明，我们提出的方法在多个基准数据集上取得了最先进的性能。源代码可在 https://github.com/Ferenas/PGSeg 获取。
英文摘要：
This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS), which learns to segment objects of arbitrary classes using mere image-text pairs. Existing works turn to enhance the vanilla vision transformer by introducing explicit grouping recognition, i.e., employing several group tokens/centroids to cluster the image tokens and perform the group-text alignment. Nevertheless, these methods suffer from a granularity inconsistency regarding the usage of group tokens, which are aligned in the all-to-one v.s. one-to-one manners during the training and inference phases, respectively. We argue that this discrepancy arises from the lack of elaborate supervision for each group token. To bridge this granularity gap, this paper explores explicit supervision for the group tokens from the prototypical knowledge. To this end, this paper proposes the non-learnable prototypical regularization (NPR) where non-learnable prototypes are estimated from source features to serve as supervision and enable contrastive matching of the group tokens. This regularization encourages the group tokens to segment objects with less redundancy and capture more comprehensive semantic regions, leading to increased compactness and richness. Based on NPR, we propose the prototypical guidance segmentation network (PGSeg) that incorporates multi-modal regularization by leveraging prototypical sources from both images and texts at different levels, progressively enhancing the segmentation capability with diverse prototypical patterns. Experimental results show that our proposed method achieves state-of-the-art performance on several benchmark datasets. The source code is available at https://github.com/Ferenas/PGSeg.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
