arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.14066cs.ROcs.AIcs.LG
收藏

KALIE:无需机器人数据微调视觉-语言模型以实现开放世界操作

KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data

University of California, Berkeley(加州大学伯克利分校);Cornell University(康奈尔大学)

Grace Tang, Swetha Rajkumar, Yifei Zhou, Homer Rich Walke, Sergey Levine, Kuan Fang

AI总结:

提出KALIE框架,通过预测基于点的可供性表示将预训练VLMs适配于机器人控制,利用人类标注的2D图像和自动数据合成流程,在无需机器人数据的情况下,仅用50个示例即可稳健解决开放世界新操作任务。

摘要:

构建通用机器人系统需要有效地赋予机器人在开放世界环境中处理新物体的能力。受大型预训练模型进展的启发,我们提出了基于想象环境的关建点可供性学习(KALIE),它以可扩展的方式将预训练的视觉语言模型(VLMs)适配于机器人控制。KALIE不直接生成运动指令,而是根据自然语言指令和场景的视觉观察,通过预测基于点的可供性表示来控制机器人。该VLM在带有由人类标注可供性的2D图像上进行训练,从而免除了在机器人系统上收集训练数据的需求。通过可供性感知的数据合成流程,KALIE基于人类手动收集的有限示例数据,自动创建大量高质量训练数据。我们证明,仅需50个示例数据点,KALIE就能学会在给定未见物体的情况下稳健地解决新操作任务。与使用预训练VLMs的基线方法相比,我们的方法持续取得更优的性能。

英文摘要:

Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting. Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained Vision Language Models (VLMs) for robotic control in a scalable manner. Instead of directly producing motor commands, KALIE controls the robot by predicting point-based affordance representations based on natural language instructions and visual observations of the scene. The VLM is trained on 2D images with affordances labeled by humans, bypassing the need for training data collected on robotic systems. Through an affordance-aware data synthesis pipeline, KALIE automatically creates massive high-quality training data based on limited example data manually collected by humans. We demonstrate that KALIE can learn to robustly solve new manipulation tasks with unseen objects given only 50 example data points. Compared to baselines using pre-trained VLMs, our approach consistently achieves superior performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑