EOV-Seg：高效开放词汇全景分割
EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentation
AI总结：
提出EOV-Seg，一种高效单阶段开放词汇全景分割框架，通过词汇感知选择和双向动态嵌入专家提升效率，在COCO训练下于ADE20K上达到24.5 PQ，推理速度比SOTA快4-19倍。
摘要：
开放词汇全景分割旨在对多样场景中的一切物体进行分割和分类，其词汇量不受限制。现有方法通常采用两阶段或单阶段框架。两阶段框架涉及使用掩码生成器生成的掩码对图像进行多次裁剪，随后进行特征提取；而单阶段框架则依赖一个重量级的掩码解码器，通过多个堆叠的Transformer块中的自注意力和交叉注意力来弥补空间位置信息的缺失。这两种方法都会产生大量的计算开销，从而阻碍模型推理的效率。为了填补效率方面的空白，我们提出了EOV-Seg，一种新颖的单阶段、共享、高效且具有空间感知能力的框架，专为开放词汇全景分割而设计。具体而言，EOV-Seg在两个方面进行了创新。首先，提出了词汇感知选择（VAS）模块，以提高视觉聚合特征的语义理解能力，并减轻掩码解码器上的特征交互负担。其次，我们引入了双向动态嵌入专家（TDEE），该模块高效利用了基于ViT的CLIP主干网络的空间感知能力。据我们所知，EOV-Seg是首个面向效率的开放词汇全景分割框架，其运行速度更快，且与最先进的方法相比性能具有竞争力。具体来说，仅使用COCO训练，EOV-Seg在ADE20K数据集上达到了24.5 PQ、32.1 mIoU和11.6 FPS，且EOV-Seg的推理时间比最先进的方法快4-19倍。特别是，配备ResNet50主干网络时，EOV-Seg在单个RTX 3090 GPU上以仅71M参数运行，达到23.8 FPS。代码可在https://github.com/nhw649/EOV-Seg获取。
英文摘要：
Open-vocabulary panoptic segmentation aims to segment and classify everything in diverse scenes across an unbounded vocabulary. Existing methods typically employ two-stage or single-stage framework. The two-stage framework involves cropping the image multiple times using masks generated by a mask generator, followed by feature extraction, while the single-stage framework relies on a heavyweight mask decoder to make up for the lack of spatial position information through self-attention and cross-attention in multiple stacked Transformer blocks. Both methods incur substantial computational overhead, thereby hindering the efficiency of model inference. To fill the gap in efficiency, we propose EOV-Seg, a novel single-stage, shared, efficient, and spatialaware framework designed for open-vocabulary panoptic segmentation. Specifically, EOV-Seg innovates in two aspects. First, a Vocabulary-Aware Selection (VAS) module is proposed to improve the semantic comprehension of visual aggregated features and alleviate the feature interaction burden on the mask decoder. Second, we introduce a Two-way Dynamic Embedding Experts (TDEE), which efficiently utilizes the spatial awareness capabilities of ViT-based CLIP backbone. To the best of our knowledge, EOV-Seg is the first open-vocabulary panoptic segmentation framework towards efficiency, which runs faster and achieves competitive performance compared with state-of-the-art methods. Specifically, with COCO training only, EOV-Seg achieves 24.5 PQ, 32.1 mIoU, and 11.6 FPS on the ADE20K dataset and the inference time of EOV-Seg is 4-19 times faster than state-of-theart methods. Especially, equipped with ResNet50 backbone, EOV-Seg runs 23.8 FPS with only 71M parameters on a single RTX 3090 GPU. Code is available at https://github.com/nhw649/EOV-Seg.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
