MOKA：通过基于标记的视觉提示实现开放世界机器人操作
MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
AI总结：
本文提出MOKA方法，通过基于标记的视觉提示将可供性推理转化为视觉问答，利用视觉语言模型解决开放世界机器人操作任务，并借助上下文学习与策略蒸馏提升性能。
摘要：
开放世界泛化要求机器人系统对物理世界和用户指令有深刻理解，以解决多样且复杂的任务。尽管视觉语言模型（VLMs）的最新进展为解决开放世界问题提供了前所未有的机会，但如何利用其能力来控制机器人仍然是一个重大挑战。本文提出了标记开放世界关键点可供性（MOKA），一种利用VLMs来解决由自由形式语言指令指定的机器人操作任务的方法。我们方法的核心是紧凑的基于点的可供性表示，它桥接了VLM对观测图像的预测与机器人在物理世界中的动作。通过提示预训练的VLM，我们的方法利用VLM从广泛数据源中获得的常识知识和概念理解来预测可供性并生成运动。为了促进VLM以零样本和少样本方式进行推理，我们提出了一种视觉提示技术，在图像上标注标记，将可供性推理转化为一系列可由VLM解决的视觉问答问题。我们进一步探索了通过上下文学习和策略蒸馏，利用MOKA收集的机器人经验来提升性能的方法。我们评估并分析了MOKA在多种桌面操作任务上的表现，包括工具使用、可变形体操作和物体重排。
英文摘要：
Open-world generalization requires robotic systems to have a profound understanding of the physical world and the user command to solve diverse and complex tasks. While the recent advancement in vision-language models (VLMs) has offered unprecedented opportunities to solve open-world problems, how to leverage their capabilities to control robots remains a grand challenge. In this paper, we introduce Marking Open-world Keypoint Affordances (MOKA), an approach that employs VLMs to solve robotic manipulation tasks specified by free-form language instructions. Central to our approach is a compact point-based representation of affordance, which bridges the VLM's predictions on observed images and the robot's actions in the physical world. By prompting the pre-trained VLM, our approach utilizes the VLM's commonsense knowledge and concept understanding acquired from broad data sources to predict affordances and generate motions. To facilitate the VLM's reasoning in zero-shot and few-shot manners, we propose a visual prompting technique that annotates marks on images, converting affordance reasoning into a series of visual question-answering problems that are solvable by the VLM. We further explore methods to enhance performance with robot experiences collected by MOKA through in-context learning and policy distillation. We evaluate and analyze MOKA's performance on various table-top manipulation tasks including tool use, deformable body manipulation, and object rearrangement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
