面向零样本全景与语义分割的可泛化语义视觉查询生成
Generalizable Semantic Vision Query Generation for Zero-shot Panoptic and Semantic Segmentation
AI总结：
本文提出 CONCAT 方法，通过条件 token 对齐和循环转换生成可泛化的语义视觉查询，在零样本全景分割上超越 SOTA，并在开放词汇语义分割中取得可比结果且测试速度提升 2 倍。
摘要：
零样本全景分割（ZPS）旨在识别训练中未包含未见类别图像的 foreground 实例和 background stuff。由于视觉数据稀疏以及从已见类别泛化到未见类别的困难，该任务仍然具有挑战性。为了更好地泛化到未见类别，我们提出了 Conditional tOken aligNment and Cycle trAnsiTion（CONCAT），以生成可泛化的语义视觉查询。首先，通过 CON 训练一个特征提取器，将视觉与语义连接起来，以提供目标查询。形式上，提出 CON 是为了将语义查询与从完整图像和掩码图像中提取的 CLIP 视觉 CLS token 对齐。为了解决未见类别缺失的问题，需要一个生成器。然而，合成伪视觉查询（即未见类别的视觉查询）的难点之一在于通过语义嵌入描述细粒度视觉细节。因此，我们采用 CAT 以语义-视觉和视觉-语义两种方式训练生成器。在语义-视觉中，提出视觉查询对比，通过将包含片段对应目标的伪视觉查询拉近、同时将不包含片段的伪视觉查询推远，来建模视觉的高粒度性。为确保生成的查询保留语义信息，在视觉-语义中，伪视觉查询被映射回语义，并由真实语义嵌入监督。在 ZPS 上的实验实现了 5.2% 的 hPQ 提升，超越了 SOTA。我们还检验了归纳式 ZPS 和开放词汇语义分割，在测试速度提高 2 倍的同时取得了可比较的结果。
英文摘要：
Zero-shot Panoptic Segmentation (ZPS) aims to recognize foreground instances and background stuff without images containing unseen categories in training. Due to the visual data sparsity and the difficulty of generalizing from seen to unseen categories, this task remains challenging. To better generalize to unseen classes, we propose Conditional tOken aligNment and Cycle trAnsiTion (CONCAT), to produce generalizable semantic vision queries. First, a feature extractor is trained by CON to link the vision and semantics for providing target queries. Formally, CON is proposed to align the semantic queries with the CLIP visual CLS token extracted from complete and masked images. To address the lack of unseen categories, a generator is required. However, one of the gaps in synthesizing pseudo vision queries, ie, vision queries for unseen categories, is describing fine-grained visual details through semantic embeddings. Therefore, we approach CAT to train the generator in semantic-vision and vision-semantic manners. In semantic-vision, visual query contrast is proposed to model the high granularity of vision by pulling the pseudo vision queries with the corresponding targets containing segments while pushing those without segments away. To ensure the generated queries retain semantic information, in vision-semantic, the pseudo vision queries are mapped back to semantic and supervised by real semantic embeddings. Experiments on ZPS achieve a 5.2% hPQ increase surpassing SOTA. We also examine inductive ZPS and open-vocabulary semantic segmentation and obtain comparative results while being 2 times faster in testing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
