arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.07062cs.CV
收藏

ThinkBot:基于思维链推理的具身指令跟随

ThinkBot: Embodied Instruction Following with Thought Chain Reasoning

Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Carnegie Mellon University(卡内基梅隆大学);Tsinghua University(清华大学)

Guanxing Lu, Ziwei Wang, Changliu Liu, Jiwen Lu, Yansong Tang

AI总结:

针对具身指令跟随中指令动作描述不连贯导致任务失败的问题,提出ThinkBot模型,通过大语言模型补全指令缺失动作、结合语义图定位交互物体,在成功率和执行效率上均优于现有最优方法。

摘要:

具身指令跟随(Embodied Instruction Following, EIF)要求智能体在复杂环境中通过与物体交互来完成人类指令。传统方法直接依据稀疏的人类指令为智能体生成动作规划,由于指令的动作描述不连贯,往往无法达成人类目标。对此,我们提出ThinkBot模型,通过推理人类指令中的思维链补全缺失的动作描述,使智能体能够遵循连贯的指令顺利完成人类目标。具体而言,我们首先设计了基于大语言模型的指令补全器,结合感知到的周围环境与已完成的子目标,补全连续人类指令之间缺失的交互物体动作。基于部分观测的场景语义图,我们提出了一种物体定位器,用于推断交互物体的位置,帮助智能体完成复杂的人类目标。在模拟环境中的大量实验表明,我们的ThinkBot在成功率和执行效率上均大幅优于当前最优的EIF方法。

英文摘要:

Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. Conventional methods directly consider the sparse human instruction to generate action plans for agents, which usually fail to achieve human goals because of the instruction incoherence in action descriptions. On the contrary, we propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions, so that the agent can successfully complete human goals by following the coherent instruction. Specifically, we first design an instruction completer based on large language models to recover the missing actions with interacted objects between consecutive human instruction, where the perceived surrounding environments and the completed sub-goals are considered for instruction completion. Based on the partially observed scene semantic maps, we present an object localizer to infer the position of interacted objects for agents to achieve complex human goals. Extensive experiments in the simulated environment show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑