KUDA：通过关键点统一动力学学习与视觉提示，用于开放词汇机器人操作
KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulation
AI总结：
本文提出 KUDA，利用关键点连接 VLM 视觉提示与神经动力学规划，将语言指令转化为可优化代价函数，提升复杂开放词汇机器人操作能力。
摘要：
随着大型语言模型（LLM）和视觉-语言模型（VLM）的快速发展，开放词汇机器人操作系统已取得显著进展。然而，许多现有方法忽视了物体动力学的重要性，限制了其在更复杂、动态任务中的适用性。在本工作中，我们提出 KUDA，这是一个通过关键点整合动力学学习与视觉提示的开放词汇操作系统，同时利用 VLM 和基于学习的神经动力学模型。我们的核心洞见是：基于关键点的目标规范既能被 VLM 理解，又能被高效转换为用于基于模型规划的代价函数。给定语言指令和视觉观测，KUDA 首先在 RGB 图像上分配关键点，并查询 VLM 以生成目标规范。随后，这些抽象的基于关键点的表示被转换为代价函数，并利用学习到的动力学模型进行优化，从而生成机器人轨迹。我们在一系列操作任务上评估 KUDA，包括跨多种物体类别的自由形式语言指令、多物体交互，以及可变形或颗粒状物体，结果证明了本框架的有效性。项目页面位于 http://kuda-dynamics.github.io。
英文摘要：
With the rapid advancement of large language models (LLMs) and vision-language models (VLMs), significant progress has been made in developing open-vocabulary robotic manipulation systems. However, many existing approaches overlook the importance of object dynamics, limiting their applicability to more complex, dynamic tasks. In this work, we introduce KUDA, an open-vocabulary manipulation system that integrates dynamics learning and visual prompting through keypoints, leveraging both VLMs and learning-based neural dynamics models. Our key insight is that a keypoint-based target specification is simultaneously interpretable by VLMs and can be efficiently translated into cost functions for model-based planning. Given language instructions and visual observations, KUDA first assigns keypoints to the RGB image and queries the VLM to generate target specifications. These abstract keypoint-based representations are then converted into cost functions, which are optimized using a learned dynamics model to produce robotic trajectories. We evaluate KUDA on a range of manipulation tasks, including free-form language instructions across diverse object categories, multi-object interactions, and deformable or granular objects, demonstrating the effectiveness of our framework. The project page is available at http://kuda-dynamics.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
