OVAL-Prompt：通过LLM可供性接地实现机器人操作的开词汇可供性定位
OVAL-Prompt: Open-Vocabulary Affordance Localization for Robot Manipulation through LLM Affordance-Grounding
AI总结：
OVAL-Prompt利用VLM和LLM实现RGB-D图像中的开词汇可供性定位，无需微调即可达到监督基线精度，支持机器人操作。项目页面：https://ekjt.github.io/OVAL-Prompt/
摘要：
为了使机器人能够有效地与物体交互，它们必须理解所遇到的每个物体的形态和功能。本质上，机器人需要理解每个物体可供哪些动作，以及这些可供性可以在何处执行。机器人最终期望在非结构化的人类环境中运行，在这种环境中，物体和可供性的集合在部署前对机器人是未知的（即开词汇设置）。在这项工作中，我们引入了OVAL-Prompt，一种基于提示的方法，用于RGB-D图像中的开词汇可供性定位。通过利用视觉语言模型（VLM）进行开词汇物体部分分割，并利用大语言模型（LLM）将每个部分分割与可供性进行接地，OVAL-Prompt展示了在无需领域特定微调的情况下对新颖物体实例、类别和可供性的泛化能力。定量实验表明，在没有任何微调的情况下，OVAL-Prompt实现了与监督基线模型竞争性的定位精度。此外，定性实验表明，OVAL-Prompt能够实现基于可供性的机器人操作，适用于开词汇物体实例和类别。项目页面：https://ekjt.github.io/OVAL-Prompt/
英文摘要：
In order for robots to interact with objects effectively, they must understand the form and function of each object they encounter. Essentially, robots need to understand which actions each object affords, and where those affordances can be acted on. Robots are ultimately expected to operate in unstructured human environments, where the set of objects and affordances is not known to the robot before deployment (i.e. the open-vocabulary setting). In this work, we introduce OVAL-Prompt, a prompt-based approach for open-vocabulary affordance localization in RGB-D images. By leveraging a Vision Language Model (VLM) for open-vocabulary object part segmentation and a Large Language Model (LLM) to ground each part-segment-affordance, OVAL-Prompt demonstrates generalizability to novel object instances, categories, and affordances without domain-specific finetuning. Quantitative experiments demonstrate that without any finetuning, OVAL-Prompt achieves localization accuracy that is competitive with supervised baseline models. Moreover, qualitative experiments show that OVAL-Prompt enables affordance-based robot manipulation of open-vocabulary object instances and categories. Project Page: https://ekjt.github.io/OVAL-Prompt/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
