展示还是讲述？有效提示视觉-语言模型进行语义分割
Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
AI总结：
本文提出PromptMatcher，通过结合文本和视觉提示提升VLMs在语义分割中的性能，实现优于现有方法的改进。
摘要：
大型视觉-语言模型（VLMs）正越来越多地被视为基础模型，可以通过提示解决多样任务，而无需特定任务训练。我们探讨了一个看似明显的问题：如何有效提示VLMs进行语义分割。为此，我们系统地评估了几种最近的模型在out-of-distribution MESS数据集集合上受文本或视觉提示引导的分割性能。我们引入了一种可扩展的提示方案，即少样本提示语义分割，受开放词汇分割和少样本学习启发。结果发现，VLMs在交并比指标上平均比专门针对特定分割任务训练的模型落后30%。此外，我们发现文本提示和视觉提示是互补的：两种模式在许多例子上都失败，而另一种可以解决。我们的分析表明，能够预知最有效的提示模式可以带来11%的性能提升。受我们的发现启发，我们提出了PromptMatcher，一种 remarkably 简单的训练免费基线，结合了文本和视觉提示，在少样本提示语义分割中取得了最先进的结果，优于最佳文本提示VLM 2.5%，以及顶级视觉提示VLM 3.5%。
英文摘要：
Large Vision-Language Models (VLMs) are increasingly being regarded as foundation models that can be instructed to solve diverse tasks by prompting, without task-specific training. We examine the seemingly obvious question: how to effectively prompt VLMs for semantic segmentation. To that end, we systematically evaluate the segmentation performance of several recent models guided by either text or visual prompts on the out-of-distribution MESS dataset collection. We introduce a scalable prompting scheme, few-shot prompted semantic segmentation, inspired by open-vocabulary segmentation and few-shot learning. It turns out that VLMs lag far behind specialist models trained for a specific segmentation task, by about 30% on average on the Intersection-over-Union metric. Moreover, we find that text prompts and visual prompts are complementary: each one of the two modes fails on many examples that the other one can solve. Our analysis suggests that being able to anticipate the most effective prompt modality can lead to a 11% improvement in performance. Motivated by our findings, we propose PromptMatcher, a remarkably simple training-free baseline that combines both text and visual prompts, achieving state-of-the-art results outperforming the best text-prompted VLM by 2.5%, and the top visual-prompted VLM by 3.5% on few-shot prompted semantic segmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
