arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.21762cs.RO
收藏

ReplanVLM:利用视觉语言模型重新规划机器人任务

ReplanVLM: Replanning Robotic Tasks with Visual Language Models

Academy for Engineering and Technology, Fudan University(复旦大学工程与技术学院)

Aoran Mei, Guo-Niu Zhu, Huaxiang Zhang, Zhongxue Gan

AI总结:

本文提出ReplanVLM框架,通过内外部错误纠正机制和重新规划策略,提升视觉语言模型在机器人任务规划中的错误纠正能力与开放世界任务成功率。

摘要:

大型语言模型(LLMs)因其在文本分析与生成方面的卓越能力以及对世界的广泛知识,在机器人任务规划中日益受到欢迎。然而,它们在解码视觉线索方面存在不足。LLMs对世界的直接感知能力有限,导致其对世界当前状态的把握不够充分。相比之下,视觉语言模型(VLMs)的出现通过集成视觉感知模块填补了这一空白,从而可以增强机器人任务规划的自主性。尽管取得了这些进展,VLMs仍面临挑战,例如即使提供了准确的指令,也可能出现任务执行错误。为了解决这些问题,本文提出了一个用于机器人任务规划的ReplanVLM框架。在本研究中,我们重点关注错误纠正干预。提出了内部错误纠正机制和外部错误纠正机制,以在相应阶段纠正错误。开发了一种重新规划策略,用于在任务执行失败时重新规划任务或纠正错误代码。在真实机器人和仿真环境中的实验结果表明,所提出的框架具有优越性,在开放世界任务中具有更高的成功率和鲁棒的错误纠正能力。我们实验的视频可在 https://youtu.be/NPk2pWKazJc 获取。

英文摘要:

Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However, they fall short in decoding visual cues. LLMs have limited direct perception of the world, which leads to a deficient grasp of the current state of the world. By contrast, the emergence of visual language models (VLMs) fills this gap by integrating visual perception modules, which can enhance the autonomy of robotic task planning. Despite these advancements, VLMs still face challenges, such as the potential for task execution errors, even when provided with accurate instructions. To address such issues, this paper proposes a ReplanVLM framework for robotic task planning. In this study, we focus on error correction interventions. An internal error correction mechanism and an external error correction mechanism are presented to correct errors under corresponding phases. A replan strategy is developed to replan tasks or correct error codes when task execution fails. Experimental results on real robots and in simulation environments have demonstrated the superiority of the proposed framework, with higher success rates and robust error correction capabilities in open-world tasks. Videos of our experiments are available at https://youtu.be/NPk2pWKazJc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑