面向两阶段开放词汇分割的提示引导掩码提议
Prompt-Guided Mask Proposal for Two-Stage Open-Vocabulary Segmentation
AI总结：
针对两阶段开放词汇分割中掩码提议易缺失目标的问题，提出提示引导掩码提议PMP方法，通过文本-查询交叉注意力生成对齐提示的掩码，在五个基准上较现有两阶段模型mIOU提升1%~3%，泛化性优异。
摘要：
我们致力于解决开放词汇分割的挑战，该任务需要以文本提示为输入，在不同环境中识别广泛类别的物体。为应对这一挑战，现有方法通常使用CLIP这类多模态模型，在共享嵌入空间中融合图像与文本特征，弥合有限词汇识别与海量词汇识别之间的差距，由此形成两阶段方案：第一阶段，掩码生成器接收输入图像生成掩码提议；第二阶段根据查询选取目标掩码。然而，生成的掩码提议中可能不存在预期的目标掩码，导致输出掩码不符合预期。在本研究中，我们提出了名为Prompt-guided Mask Proposal（提示引导掩码提议，简称PMP）的新方法，掩码生成器接收输入文本提示，并在这些提示的引导下生成掩码。与无输入提示生成的掩码提议相比，PMP生成的掩码与输入提示的对齐度更高。为实现PMP，我们设计了文本令牌与查询令牌之间的交叉注意力机制，该机制可在每次解码后生成提示引导的掩码提议。我们将PMP与多个采用基于查询的分割骨干网络的现有方法相结合，在五个基准数据集上的实验证明了该方法的有效性，相比当前两阶段模型实现了显著提升（以mIOU计，绝对性能提升1%~3%）。在这些基准上的稳定性能提升表明，我们提出的轻量级提示感知方法具备有效的泛化能力。
英文摘要：
We tackle the challenge of open-vocabulary segmentation, where we need to identify objects from a wide range of categories in different environments, using text prompts as our input. To overcome this challenge, existing methods often use multi-modal models like CLIP, which combine image and text features in a shared embedding space to bridge the gap between limited and extensive vocabulary recognition, resulting in a two-stage approach: In the first stage, a mask generator takes an input image to generate mask proposals, and the in the second stage the target mask is picked based on the query. However, the expected target mask may not exist in the generated mask proposals, which leads to an unexpected output mask. In our work, we propose a novel approach named Prompt-guided Mask Proposal (PMP) where the mask generator takes the input text prompts and generates masks guided by these prompts. Compared with mask proposals generated without input prompts, masks generated by PMP are better aligned with the input prompts. To realize PMP, we designed a cross-attention mechanism between text tokens and query tokens which is capable of generating prompt-guided mask proposals after each decoding. We combined our PMP with several existing works employing a query-based segmentation backbone and the experiments on five benchmark datasets demonstrate the effectiveness of this approach, showcasing significant improvements over the current two-stage models (1% ~ 3% absolute performance gain in terms of mIOU). The steady improvement in performance across these benchmarks indicates the effective generalization of our proposed lightweight prompt-aware method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
