Mask-Adapter：开放词汇分割中掩码才是关键
Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation
AI总结：
本文提出Mask-Adapter，通过从候选掩码提取语义激活图并引入掩码一致性损失，解决开放词汇分割中掩码池化的分类性能局限，以即插即用方式提升多种方法的零样本分割精度。
摘要：
近年来，开放词汇分割方法采用掩码生成器预测分割掩码，并利用预训练的视觉-语言模型（如CLIP）通过掩码池化对这些掩码进行分类。尽管这些方法展现出良好的效果，但一个反直觉的现象是：准确的掩码往往无法通过在掩码区域内池化CLIP图像嵌入来获得准确的分类结果。本文揭示了掩码池化的性能局限，并提出Mask-Adapter，一种简单而有效的方法来应对开放词汇分割中的这些挑战。与直接使用候选掩码相比，我们提出的Mask-Adapter从候选掩码中提取语义激活图，提供更丰富的上下文信息，并确保掩码与CLIP之间的对齐。此外，我们提出了一种掩码一致性损失，鼓励具有相似IoU的候选掩码获得相似的CLIP嵌入，以增强模型对不同预测掩码的鲁棒性。Mask-Adapter以即插即用的方式无缝集成到基于掩码池化的开放词汇分割方法中，提供更准确的分类结果。在多个零样本基准上的大量实验表明，所提出的Mask-Adapter在多个成熟方法上带来了显著的性能提升。值得注意的是，Mask-Adapter还能有效扩展到SAM，并在多个开放词汇分割数据集上取得了令人印象深刻的结果。代码和模型可在https://github.com/hustvl/MaskAdapter获取。
英文摘要：
Recent open-vocabulary segmentation methods adopt mask generators to predict segmentation masks and leverage pre-trained vision-language models, e.g., CLIP, to classify these masks via mask pooling. Although these approaches show promising results, it is counterintuitive that accurate masks often fail to yield accurate classification results through pooling CLIP image embeddings within the mask regions. In this paper, we reveal the performance limitations of mask pooling and introduce Mask-Adapter, a simple yet effective method to address these challenges in open-vocabulary segmentation. Compared to directly using proposal masks, our proposed Mask-Adapter extracts semantic activation maps from proposal masks, providing richer contextual information and ensuring alignment between masks and CLIP. Additionally, we propose a mask consistency loss that encourages proposal masks with similar IoUs to obtain similar CLIP embeddings to enhance models' robustness to varying predicted masks. Mask-Adapter integrates seamlessly into open-vocabulary segmentation methods based on mask pooling in a plug-and-play manner, delivering more accurate classification results. Extensive experiments across several zero-shot benchmarks demonstrate significant performance gains for the proposed Mask-Adapter on several well-established methods. Notably, Mask-Adapter also extends effectively to SAM and achieves impressive results on several open-vocabulary segmentation datasets. Code and models are available at https://github.com/hustvl/MaskAdapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
