用于小样本分割及更多任务的视觉与文本先验引导掩码组装
Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyond
AI总结：
针对基于 CLIP 的小样本分割存在基类偏置的问题，本文提出 PGMA-Net，通过类别无关先验、掩码组装模块和通道丢弃分层解码器实现统一的任意样本分割，并取得最先进性能。
摘要：
小样本分割（FSS）旨在利用少量带标注图像分割新类别。由于 CLIP 在对齐视觉与文本信息方面的优势，整合 CLIP 可以增强 FSS 模型的泛化能力。然而，即使使用 CLIP 模型，现有基于 CLIP 的 FSS 方法仍会受到面向基类的有偏预测影响，这种偏差由类别特定的特征层级交互造成。为解决该问题，我们提出了视觉与文本先验引导掩码组装网络（PGMA-Net）。它采用类别无关的掩码组装过程来缓解偏差，并通过亲和度组装先验，将多样化任务以统一方式形式化。具体而言，首先将类别相关的文本和视觉特征转换为概率图形式的类别无关先验。随后，引入包含多个通用组装单元（GAU）的先验引导掩码组装模块（PGMAM）。该模块考虑多样化且即插即用的交互，例如视觉-文本交互、图像间与图像内交互、免训练交互以及高阶交互。最后，为确保类别无关能力，提出了带通道丢弃机制的分层解码器（HDCDM），以灵活利用组装后的掩码和低层特征，而不依赖任何类别特定信息。该方法在 FSS 任务上取得了新的最先进结果，在 1-shot 场景下，于 PASCAL-5^i 上达到 77.6 的 mIoU，在 COCO-20^i 上达到 59.4 的 mIoU。除此之外，我们表明，无需额外重新训练，所提出的 PGMA-Net 即可解决 bbox 级和跨域 FSS、协同分割、零样本分割（ZSS）任务，从而形成一个任意样本分割框架。
英文摘要：
Few-shot segmentation (FSS) aims to segment the novel classes with a few annotated images. Due to CLIP's advantages of aligning visual and textual information, the integration of CLIP can enhance the generalization ability of FSS model. However, even with the CLIP model, the existing CLIP-based FSS methods are still subject to the biased prediction towards base classes, which is caused by the class-specific feature level interactions. To solve this issue, we propose a visual and textual Prior Guided Mask Assemble Network (PGMA-Net). It employs a class-agnostic mask assembly process to alleviate the bias, and formulates diverse tasks into a unified manner by assembling the prior through affinity. Specifically, the class-relevant textual and visual features are first transformed to class-agnostic prior in the form of probability map. Then, a Prior-Guided Mask Assemble Module (PGMAM) including multiple General Assemble Units (GAUs) is introduced. It considers diverse and plug-and-play interactions, such as visual-textual, inter- and intra-image, training-free, and high-order ones. Lastly, to ensure the class-agnostic ability, a Hierarchical Decoder with Channel-Drop Mechanism (HDCDM) is proposed to flexibly exploit the assembled masks and low-level features, without relying on any class-specific information. It achieves new state-of-the-art results in the FSS task, with mIoU of $77.6$ on $\text{PASCAL-}5^i$ and $59.4$ on $\text{COCO-}20^i$ in 1-shot scenario. Beyond this, we show that without extra re-training, the proposed PGMA-Net can solve bbox-level and cross-domain FSS, co-segmentation, zero-shot segmentation (ZSS) tasks, leading an any-shot segmentation framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
