arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.02405cs.ROcs.CV
收藏

提升语言条件机器人操作泛化能力

Improving Generalization of Language-Conditioned Robot Manipulation

Centre for Intelligent Sensing, Queen Mary University of London(智能传感中心,伦敦女王玛丽大学);Idiap Research Institute and École Polytechnique Fédérale de Lausanne(Idiap研究机构和日内瓦联邦理工学院)

Chenglin Cui, Chaoran Zhu, Changjae Oh, Andrea Cavallaro

AI总结:

本文提出一个两阶段框架,通过实例级语义融合模块,仅用少量演示微调视觉语言模型,提升语言条件机器人操作在未见环境中的泛化能力,并实现真实场景零样本操作。

摘要:

机器人操作任务的控制通常依赖于视觉输入。视觉语言模型(VLMs)的最新进展使得利用自然语言指令来调节视觉输入并控制机器人在更广泛的环境中成为可能。然而,现有方法需要大量数据来微调VLM,以便在未见环境中运行。在本文中,我们提出了一个框架,该框架仅从少量演示中学习物体排列任务。我们提出了一个两阶段框架,将物体排列任务划分为目标定位阶段(用于拾取物体)和区域确定阶段(用于放置物体)。我们提出了一个实例级语义融合模块,该模块将实例级图像裁剪与文本嵌入对齐,使模型能够识别由自然语言指令定义的目标物体。我们在仿真和真实机器人环境中验证了我们的方法。我们的方法通过少量演示进行微调,提高了泛化能力,并在真实机器人操作场景中展示了零样本能力。

英文摘要:

The control of robots for manipulation tasks generally relies on visual input. Recent advances in vision-language models (VLMs) enable the use of natural language instructions to condition visual input and control robots in a wider range of environments. However, existing methods require a large amount of data to fine-tune VLMs for operating in unseen environments. In this paper, we present a framework that learns object-arrangement tasks from just a few demonstrations. We propose a two-stage framework that divides object-arrangement tasks into a target localization stage, for picking the object, and a region determination stage for placing the object. We present an instance-level semantic fusion module that aligns the instance-level image crops with the text embedding, enabling the model to identify the target objects defined by the natural language instructions. We validate our method on both simulation and real-world robotic environments. Our method, fine-tuned with a few demonstrations, improves generalization capability and demonstrates zero-shot ability in real-robot manipulation scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑