PIVOT：迭代视觉提示为VLM引出可操作知识
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
AI总结：
本文提出PIVOT方法，通过迭代视觉提示将任务转化为视觉问答，使VLMs无需微调即可零样本输出空间坐标和动作，实现机器人控制与导航。
摘要：
视觉语言模型（VLMs）在各种任务中展现出了令人印象深刻的能力，从逻辑推理到视觉理解。这为与世界进行更丰富的交互打开了大门，例如机器人控制。然而，VLMs只能产生文本输出，而机器人控制和其他空间任务需要输出连续的坐标、动作或轨迹。我们如何在不针对特定任务数据进行微调的情况下，使VLMs能够处理此类场景？在本文中，我们提出了一种名为Prompting with Iterative Visual Optimization (PIVOT)的VLMs新型视觉提示方法，该方法将任务转化为迭代式的视觉问答。在每次迭代中，图像上会标注VLM可以参考的提议视觉表示（例如，候选机器人动作、定位或轨迹）。然后，VLM为任务选择最佳提议。这些提议被迭代细化，使VLM最终能够锁定最佳可用答案。我们在真实世界机器人导航、基于图像的真实世界操作、模拟中的指令遵循以及定位等其他空间推理任务上研究了PIVOT。我们发现，也许令人惊讶的是，我们的方法能够在没有任何机器人训练数据的情况下实现机器人系统的零样本控制，以及在各种环境中导航和其他能力。尽管当前性能远非完美，但我们的工作突出了这一新范式的潜力和局限性，并展示了Internet-Scale VLMs在机器人和空间推理领域的一种有前景的方法。网站：pivot-prompt.github.io和HuggingFace：https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo。
英文摘要：
Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. This opens the door to richer interaction with the world, for example robotic control. However, VLMs produce only textual outputs, while robotic control and other spatial tasks require outputting continuous coordinates, actions, or trajectories. How can we enable VLMs to handle such settings without fine-tuning on task-specific data? In this paper, we propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as iterative visual question answering. In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e.g., candidate robot actions, localizations, or trajectories). The VLM then selects the best ones for the task. These proposals are iteratively refined, allowing the VLM to eventually zero in on the best available answer. We investigate PIVOT on real-world robotic navigation, real-world manipulation from images, instruction following in simulation, and additional spatial inference tasks such as localization. We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities. Although current performance is far from perfect, our work highlights potentials and limitations of this new regime and shows a promising approach for Internet-Scale VLMs in robotic and spatial reasoning domains. Website: pivot-prompt.github.io and HuggingFace: https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
