PosSAM: 全景开放词汇分割任意模型
PosSAM: Panoptic Open-vocabulary Segment Anything
AI总结：
PosSAM是一个端到端开放词汇全景分割模型，融合SAM与CLIP，通过LDP模块和MASE算法实现无偏分类与掩码增强，在COCO和ADE20K数据集上分别超越SOTA 2.4和4.6 PQ。
摘要：
在本文中，我们提出了一种开放词汇全景分割模型，该模型在端到端框架内有效统一了分割任意模型（SAM）与视觉-语言CLIP模型的优势。尽管SAM在生成空间感知掩码方面表现出色，但其解码器在识别对象类别信息方面存在不足，且在没有额外指导的情况下倾向于过度分割。现有方法通过采用多阶段技术并使用单独模型生成类别感知提示（如边界框或分割掩码）来解决这一限制。我们提出的方法PosSAM是一个端到端模型，它利用SAM的空间丰富特征生成实例感知掩码，并利用CLIP的语义判别特征进行有效的实例分类。具体而言，我们解决了SAM的局限性，并提出了一种新颖的局部判别池化（LDP）模块，利用类别无关的SAM和类别感知的CLIP特征进行无偏的开放词汇分类。此外，我们引入了一种掩码感知选择性集成（MASE）算法，该算法在推理过程中自适应地提升生成掩码的质量，并增强每张图像的开放词汇分类性能。我们进行了大量实验，证明了我们的方法在多个数据集上具有很强的泛化能力，取得了最先进的性能，相较于最先进的开放词汇全景分割方法有显著提升。在COCO到ADE20K和ADE20K到COCO两种设置下，PosSAM分别以2.4 PQ和4.6 PQ的大幅优势超越了之前的最先进方法。项目网站：https://vibashan.github.io/possam-web/。
英文摘要：
In this paper, we introduce an open-vocabulary panoptic segmentation model that effectively unifies the strengths of the Segment Anything Model (SAM) with the vision-language CLIP model in an end-to-end framework. While SAM excels in generating spatially-aware masks, it's decoder falls short in recognizing object class information and tends to oversegment without additional guidance. Existing approaches address this limitation by using multi-stage techniques and employing separate models to generate class-aware prompts, such as bounding boxes or segmentation masks. Our proposed method, PosSAM is an end-to-end model which leverages SAM's spatially rich features to produce instance-aware masks and harnesses CLIP's semantically discriminative features for effective instance classification. Specifically, we address the limitations of SAM and propose a novel Local Discriminative Pooling (LDP) module leveraging class-agnostic SAM and class-aware CLIP features for unbiased open-vocabulary classification. Furthermore, we introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image. We conducted extensive experiments to demonstrate our methods strong generalization properties across multiple datasets, achieving state-of-the-art performance with substantial improvements over SOTA open-vocabulary panoptic segmentation methods. In both COCO to ADE20K and ADE20K to COCO settings, PosSAM outperforms the previous state-of-the-art methods by a large margin, 2.4 PQ and 4.6 PQ, respectively. Project Website: https://vibashan.github.io/possam-web/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
