基于大模型学习具身机器人任务的指令引导操作可供性
Learning Instruction-Guided Manipulation Affordance via Large Models for Embodied Robotic Tasks
AI总结：
针对语言指令无法动态改变操作区域的问题，提出IGANet利用预训练视觉语言编码器预测指令引导的可供性图，结合VLMs数据增强与LLMs执行动作，在真实实验中表现更优且泛化能力更强。
摘要：
我们研究语言指令引导的机器人操作任务，其中具身机器人需根据语言指令操作目标物体。在以往的研究中，预测的目标物体操作区域通常不会随语言指令的说明而改变，这意味着语言感知与操作预测是分离的。然而，在人类行为模式中，同一物体的操作区域会因不同的语言指令而改变。在本文中，我们提出Instruction-Guided Affordance Net（IGANet），通过利用在大规模数据集上预训练的视觉和语言编码器的强大先验，来预测指令引导机器人操作任务的可供性图。我们开发了基于Vision-Language-Models(VLMs)的数据增强流水线，可自动生成大量数据用于模型训练。此外，借助Large-Language-Models(LLMs)，动作可以被有效执行以完成指令定义的任务。一系列真实世界实验表明，我们的方法利用生成的数据能取得更好的性能。此外，我们的模型能更好地泛化到包含未见物体和语言指令的场景。
英文摘要：
We study the task of language instruction-guided robotic manipulation, in which an embodied robot is supposed to manipulate the target objects based on the language instructions. In previous studies, the predicted manipulation regions of the target object typically do not change with specification from the language instructions, which means that the language perception and manipulation prediction are separate. However, in human behavioral patterns, the manipulation regions of the same object will change for different language instructions. In this paper, we propose Instruction-Guided Affordance Net (IGANet) for predicting affordance maps of instruction-guided robotic manipulation tasks by utilizing powerful priors from vision and language encoders pre-trained on large-scale datasets. We develop a Vison-Language-Models(VLMs)-based data augmentation pipeline, which can generate a large amount of data automatically for model training. Besides, with the help of Large-Language-Models(LLMs), actions can be effectively executed to finish the tasks defined by instructions. A series of real-world experiments revealed that our method can achieve better performance with generated data. Moreover, our model can generalize better to scenarios with unseen objects and language instructions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
