细粒度视觉提示
Fine-Grained Visual Prompting
AI总结：
本文提出 FGVP 零样本框架，利用通用分割模型的像素级掩码生成细粒度视觉提示，并以 Blur Reverse Mask 显著提升 RefCOCO 系列指代表达理解性能。
摘要：
诸如 CLIP 之类的视觉-语言模型（VLM）在图像级视觉感知中展现出令人瞩目的零样本迁移能力。然而，这些模型在需要精确定位和识别的实例级任务上表现有限。以往研究表明，加入彩色框或圆圈等视觉提示可以提升模型识别感兴趣对象的能力。尽管如此，与语言提示相比，视觉提示设计仍很少被探索。现有方法采用彩色框或圆圈等粗糙视觉线索，往往因包含无关且带噪声的像素而导致次优性能。本文中，我们通过研究分割掩码及其变体等更细粒度的标记，仔细考察视觉提示设计。此外，我们提出一种新的零样本框架，利用从通用分割模型获得的像素级标注来进行细粒度视觉提示。研究发现，在目标掩码外部直接施加模糊（称为 Blur Reverse Mask）具有异常出色的效果。所提出的提示策略利用精确掩码标注，减少对弱相关区域的关注，同时保留目标与周围背景之间的空间连贯性。我们的 Fine-Grained Visual Prompting（FGVP）在 RefCOCO、RefCOCO+ 和 RefCOCOg 基准上的零样本指代表达理解中表现出优越性能。它较以往方法平均领先 3.0% 至 4.6%，在 RefCOCO+ testA 子集上的最大提升达 12.5%。代码位于 https://github.com/ylingfeng/FGVP。
英文摘要：
Vision-Language Models (VLMs), such as CLIP, have demonstrated impressive zero-shot transfer capabilities in image-level visual perception. However, these models have shown limited performance in instance-level tasks that demand precise localization and recognition. Previous works have suggested that incorporating visual prompts, such as colorful boxes or circles, can improve the ability of models to recognize objects of interest. Nonetheless, compared to language prompting, visual prompting designs are rarely explored. Existing approaches, which employ coarse visual cues such as colorful boxes or circles, often result in sub-optimal performance due to the inclusion of irrelevant and noisy pixels. In this paper, we carefully study the visual prompting designs by exploring more fine-grained markings, such as segmentation masks and their variations. In addition, we introduce a new zero-shot framework that leverages pixel-level annotations acquired from a generalist segmentation model for fine-grained visual prompting. Consequently, our investigation reveals that a straightforward application of blur outside the target mask, referred to as the Blur Reverse Mask, exhibits exceptional effectiveness. This proposed prompting strategy leverages the precise mask annotations to reduce focus on weakly related regions while retaining spatial coherence between the target and the surrounding background. Our Fine-Grained Visual Prompting (FGVP) demonstrates superior performance in zero-shot comprehension of referring expressions on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks. It outperforms prior methods by an average margin of 3.0% to 4.6%, with a maximum improvement of 12.5% on the RefCOCO+ testA subset. Code is available at https://github.com/ylingfeng/FGVP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
