RoboPoint：一种用于机器人空间可供性预测的视觉-语言模型
RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
AI总结：
本文提出RoboPoint视觉-语言模型，通过自动合成数据生成流程进行指令微调，无需真实数据即可预测图像关键点可供性，在空间可供性预测准确率和下游任务成功率上显著超越GPT-4o和PIVOT。
摘要：
从重新摆放桌面物体到将杂货放入货架，机器人必须规划精确的动作点才能准确可靠地执行任务。尽管近期采用了视觉-语言模型（VLMs）来控制机器人行为，但VLMs在利用语言精确表达机器人动作方面仍存在困难。我们提出了一种自动合成数据生成流程，通过指令微调使VLMs适应机器人领域及其需求。利用该流程，我们训练了RoboPoint，这是一种在给定语言指令的情况下预测图像关键点可供性的VLM。与替代方法相比，我们的方法无需真实世界数据采集或人类演示，因此对多样化环境和视角具有更强的可扩展性。此外，RoboPoint是一个通用模型，能够支持多种下游应用，如机器人导航、操作和增强现实（AR）辅助。我们的实验表明，RoboPoint在空间可供性预测准确率上比最先进的VLMs（GPT-4o）和视觉提示技术（PIVOT）高出21.8%，在下游任务成功率上高出30.5%。项目网站：https://robo-point.github.io。
英文摘要：
From rearranging objects on a table to putting groceries into shelves, robots must plan precise action points to perform tasks accurately and reliably. In spite of the recent adoption of vision language models (VLMs) to control robot behavior, VLMs struggle to precisely articulate robot actions using language. We introduce an automatic synthetic data generation pipeline that instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Compared to alternative approaches, our method requires no real-world data collection or human demonstration, making it much more scalable to diverse environments and viewpoints. In addition, RoboPoint is a general model that enables several downstream applications such as robot navigation, manipulation, and augmented reality (AR) assistance. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks. Project website: https://robo-point.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
