NaturalVLM：利用细粒度自然语言进行可供性引导的视觉操作
NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation
AI总结：
针对家庭机器人多步操作难题，提出NrVLM基准（15个任务、4500+片段）及基于细粒度语言指令逐步执行的学习框架，通过动作与感知提示实现跨模态对齐，输出接触点与末端姿态，实验验证有效。
摘要：
让家庭辅助机器人能够根据人类语言指令感知并操作各种3D物体是一项关键挑战。以往的研究主要集中于简单且面向任务的指令，例如“滑开顶部抽屉”。然而，许多现实世界的任务需要复杂的多步推理，在没有人类指令的情况下，这些任务对机器人操作来说将变得极其困难。为了应对这些挑战，我们引入了一个全面的基准测试NrVLM，包含15个不同的操作任务，超过4500个带有细粒度语言指令的详细标注片段。我们将长期任务过程拆分为多个步骤，每个步骤都有一条自然语言指令。此外，我们提出了一种新颖的学习框架，根据细粒度指令逐步完成操作任务。具体来说，我们首先识别要执行的指令，同时考虑视觉观察和末端执行器的当前状态。随后，我们的方法通过动作提示和感知提示促进显式学习，以促进操作感知的跨模态对齐。利用视觉观察和语言指导，我们的模型输出一系列可操作的操作预测，包括接触点和末端执行器姿态。我们使用所提出的基准测试NrVLM评估了我们的方法和基线。实验结果表明了我们方法的有效性。更多细节，请参阅https://sites.google.com/view/naturalvlm。
英文摘要：
Enabling home-assistant robots to perceive and manipulate a diverse range of 3D objects based on human language instructions is a pivotal challenge. Prior research has predominantly focused on simplistic and task-oriented instructions, i.e., "Slide the top drawer open". However, many real-world tasks demand intricate multi-step reasoning, and without human instructions, these will become extremely difficult for robot manipulation. To address these challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15 distinct manipulation tasks, containing over 4500 episodes meticulously annotated with fine-grained language instructions. We split the long-term task process into several steps, with each step having a natural language instruction. Moreover, we propose a novel learning framework that completes the manipulation task step-by-step according to the fine-grained instructions. Specifically, we first identify the instruction to execute, taking into account visual observations and the end-effector's current state. Subsequently, our approach facilitates explicit learning through action-prompts and perception-prompts to promote manipulation-aware cross-modality alignment. Leveraging both visual observations and linguistic guidance, our model outputs a sequence of actionable predictions for manipulation, including contact points and end-effector poses. We evaluate our method and baselines using the proposed benchmark NrVLM. The experimental results demonstrate the effectiveness of our approach. For additional details, please refer to https://sites.google.com/view/naturalvlm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
