DoReMi:通过检测并从规划-执行偏差中恢复实现语言模型的具身落地
DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment
AI总结:
针对机器人任务中LLM生成的高层规划与底层执行易出现偏差的问题,提出DoReMi框架,通过LLM生成规划与约束、VLM持续检测偏差并及时恢复,提升了任务成功率并缩短了完成时间。
摘要:
大语言模型(LLMs)编码了海量语义知识,具备出色的理解与推理能力。已有研究探索了如何将LLMs应用于机器人任务以生成可行、可执行的文本规划。但由于环境扰动或控制器设计不完善,物理世界中的底层执行可能会偏离高层文本规划。本文提出了一种新型语言模型具身落地框架**DoReMi**,可即时检测规划与执行之间的偏差并从中恢复。具体而言,我们利用LLMs发挥双重作用:不仅辅助高层规划,还生成可指示执行过程中偏差的约束条件。随后使用视觉语言模型(VLMs)持续检测约束违反情况。我们的流水线可监控底层执行,一旦出现规划-执行偏差即可及时恢复。在包含机械臂与人形机器人的各类复杂任务上的实验表明,我们的方法能带来更高的任务成功率与更短的任务完成时间。DoReMi的演示视频可在https://sites.google.com/view/doremi-paper查看。
英文摘要:
Large language models (LLMs) encode a vast amount of semantic knowledge and possess remarkable understanding and reasoning capabilities. Previous work has explored how to ground LLMs in robotic tasks to generate feasible and executable textual plans. However, low-level execution in the physical world may deviate from the high-level textual plan due to environmental perturbations or imperfect controller design. In this paper, we propose \textbf{DoReMi}, a novel language model grounding framework that enables immediate Detection and Recovery from Misalignments between plan and execution. Specifically, we leverage LLMs to play a dual role, aiding not only in high-level planning but also generating constraints that can indicate misalignment during execution. Then vision language models (VLMs) are utilized to detect constraint violations continuously. Our pipeline can monitor the low-level execution and enable timely recovery if certain plan-execution misalignment occurs. Experiments on various complex tasks including robot arms and humanoid robots demonstrate that our method can lead to higher task success rates and shorter task completion times. Videos of DoReMi are available at \url{https://sites.google.com/view/doremi-paper}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。