SAGE：桥接语义部件与可操作部件以实现铰接物体的可泛化操控

中文摘要 AI 辅助

为了与日常生活中结构和功能多样的铰接物体进行交互，理解物体部件在用户指令理解和任务执行中都起着核心作用。然而，部件的语义含义与其物理功能之间可能存在的不一致，给设计通用系统带来了挑战。为了解决这一问题，我们提出了 SAGE，这是一个新颖的框架，它桥接铰接物体的语义部件和可操作部件，从而在自然语言指令下实现可泛化的操控。具体而言，给定一个铰接物体，我们首先观察其上的所有语义部件，在此基础上，指令解释器提出可能的动作程序，将自然语言指令具体化。然后，一个部件接地模块将语义部件映射为所谓的可泛化可操作部件（GAParts），这些部件固有地携带部件运动信息。末端执行器轨迹在 GAParts 上进行预测，并与动作程序一起构成可执行策略。此外，还引入了一个交互式反馈模块来响应失败，从而闭合回路并提高整个框架的鲁棒性。我们框架成功的关键在于大型视觉语言模型（VLM）与小型领域专用模型之间的联合提议和知识融合，用于上下文理解和部件感知，前者提供一般性直觉，后者充当专家事实。仿真和真实机器人实验均表明，我们在处理大量具有多样语言指令目标的铰接物体方面具有有效性。

英文摘要

To interact with daily-life articulated objects of diverse structures and functionalities, understanding the object parts plays a central role in both user instruction comprehension and task execution. However, the possible discordance between the semantic meaning and physics functionalities of the parts poses a challenge for designing a general system. To address this problem, we propose SAGE, a novel framework that bridges semantic and actionable parts of articulated objects to achieve generalizable manipulation under natural language instructions. More concretely, given an articulated object, we first observe all the semantic parts on it, conditioned on which an instruction interpreter proposes possible action programs that concretize the natural language instruction. Then, a part-grounding module maps the semantic parts into so-called Generalizable Actionable Parts (GAParts), which inherently carry information about part motion. End-effector trajectories are predicted on the GAParts, which, together with the action program, form an executable policy. Additionally, an interactive feedback module is incorporated to respond to failures, which closes the loop and increases the robustness of the overall framework. Key to the success of our framework is the joint proposal and knowledge fusion between a large vision-language model (VLM) and a small domain-specific model for both context comprehension and part perception, with the former providing general intuitions and the latter serving as expert facts. Both simulation and real-robot experiments show our effectiveness in handling a large variety of articulated objects with diverse language-instructed goals.

发表机构

• Stanford University（斯坦福大学）
• Peking University（北京大学）
• Beijing Academy of Artificial Intelligence（北京人工智能研究院）

机构由 AI 辅助整理，请以论文原文为准。