CLIP 对红色圆圈了解多少？面向 VLM 的视觉提示工程
What does CLIP know about a red circle? Visual prompt engineering for VLMs
AI总结：
本文提出面向 VLM 的视觉提示工程，发现 CLIP 可通过红色圆圈引导视觉注意并保留全局信息，从而提升零样本指代表达理解与关键点定位性能。
摘要：
诸如 CLIP 之类的大规模视觉-语言模型学习到了强大的图文表征，并已在从零样本分类到文本生成图像等众多应用中得到使用。尽管如此，它们通过提示解决新型判别任务的能力仍落后于 GPT-3 等大型语言模型。本文探索用于解决分类之外计算机视觉任务的视觉提示工程思想，即在图像空间而非文本中进行编辑。具体而言，我们发现 CLIP 的一种涌现能力：只需在对象周围绘制一个红色圆圈，就可以将模型注意力引导至该区域，同时保留全局信息。我们展示了这一简单方法的能力：它在零样本指代表达理解中达到最先进水平，并在关键点定位任务中取得强劲性能。最后，我们提醒关注大型语言-视觉模型的一些潜在伦理问题。
英文摘要：
Large-scale Vision-Language Models, such as CLIP, learn powerful image-text representations that have found numerous applications, from zero-shot classification to text-to-image generation. Despite that, their capabilities for solving novel discriminative tasks via prompting fall behind those of large language models, such as GPT-3. Here we explore the idea of visual prompt engineering for solving computer vision tasks beyond classification by editing in image space instead of text. In particular, we discover an emergent ability of CLIP, where, by simply drawing a red circle around an object, we can direct the model's attention to that region, while also maintaining global information. We show the power of this simple approach by achieving state-of-the-art in zero-shot referring expressions comprehension and strong performance in keypoint localization tasks. Finally, we draw attention to some potential ethical concerns of large language-vision models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
