基于运动学感知提示的大语言模型通用关节物体操作
Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs
AI总结：
本文提出运动学感知提示框架，将关节物体结构解析为统一文本描述，通过思维链提示大语言模型生成低层3D操作路径点，在16类48个实例上验证了优于传统方法的表现及对未见类别的零样本泛化能力。
摘要：
通用关节物体操作对于家庭辅助机器人至关重要。近期研究主要集中于从演示中进行模仿学习或在仿真中进行强化学习，然而由于真实世界数据采集和精确物体仿真的高昂成本，这些工作在多样关节物体上的广泛适应性仍然面临挑战。近年来，许多工作尝试利用大语言模型（LLMs）强大的上下文学习能力来实现通用机器人操作，但其中大多数研究聚焦于高层任务规划，忽略了低层机器人控制。在本工作中，基于物体的运动学结构决定我们如何操作它的思想，我们提出了一种运动学感知提示框架，该框架利用物体的运动学知识提示LLMs生成低层运动轨迹路径点，从而支持各种物体操作。为了有效地向LLMs提示不同物体的运动学结构，我们设计了一个统一的运动学知识解析器，将各种关节物体表示为包含运动学关节和接触位置的统一文本描述。在此统一描述的基础上，我们提出了一种运动学感知规划器模型，通过设计的运动学感知思维链提示方法生成精确的3D操作路径点。我们的评估涵盖了16个不同类别中的48个实例，结果表明我们的框架不仅在8个已见类别上优于传统方法，而且对8个未见关节物体类别展现出强大的零样本能力。此外，在7个不同物体类别上的真实世界实验证明了我们框架在实际场景中的适应性。代码已发布在https://github.com/GeWu-Lab/LLM_articulated_object_manipulation/tree/main。
英文摘要：
Generalizable articulated object manipulation is essential for home-assistant robots. Recent efforts focus on imitation learning from demonstrations or reinforcement learning in simulation, however, due to the prohibitive costs of real-world data collection and precise object simulation, it still remains challenging for these works to achieve broad adaptability across diverse articulated objects. Recently, many works have tried to utilize the strong in-context learning ability of Large Language Models (LLMs) to achieve generalizable robotic manipulation, but most of these researches focus on high-level task planning, sidelining low-level robotic control. In this work, building on the idea that the kinematic structure of the object determines how we can manipulate it, we propose a kinematic-aware prompting framework that prompts LLMs with kinematic knowledge of objects to generate low-level motion trajectory waypoints, supporting various object manipulation. To effectively prompt LLMs with the kinematic structure of different objects, we design a unified kinematic knowledge parser, which represents various articulated objects as a unified textual description containing kinematic joints and contact location. Building upon this unified description, a kinematic-aware planner model is proposed to generate precise 3D manipulation waypoints via a designed kinematic-aware chain-of-thoughts prompting method. Our evaluation spanned 48 instances across 16 distinct categories, revealing that our framework not only outperforms traditional methods on 8 seen categories but also shows a powerful zero-shot capability for 8 unseen articulated object categories. Moreover, the real-world experiments on 7 different object categories prove our framework's adaptability in practical scenarios. Code is released at https://github.com/GeWu-Lab/LLM_articulated_object_manipulation/tree/main.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
