基于大语言模型的具身任务规划
Embodied Task Planning with Large Language Models
AI总结：
本文提出TaPA框架，通过将大语言模型与视觉感知模型对齐并利用多模态数据集进行接地规划微调，使具身智能体能够根据场景中已有物体生成可执行计划，在复杂环境中显著提升了任务规划成功率。
摘要：
为具身智能体配备常识对于机器人在通用环境中成功完成复杂人类指令至关重要。近期的大语言模型（LLM）能够在复杂任务的规划生成中为智能体嵌入丰富的语义知识，但它们缺乏关于现实世界的信息，通常会产生不可执行的动作序列。在本文中，我们提出了一种用于具身任务的TAsk Planing Agent（TaPA），在物理场景约束下进行接地规划，其中智能体通过将LLM与视觉感知模型对齐，根据场景中已存在的物体生成可执行的计划。具体来说，我们首先构建了一个包含室内场景、指令和动作计划三元组的多模态数据集，其中我们为GPT-3.5提供设计好的提示词和场景中已有物体的列表，以生成大量指令及相应的规划动作。生成的数据被用于对预训练LLM进行接地规划微调。在推理过程中，我们通过将开放词汇物体检测器扩展到在不同可达位置采集的多视角RGB图像来发现场景中的物体。实验结果表明，我们的TaPA框架生成的计划在成功率上显著超过LLaVA和GPT-3.5，这表明具身任务规划在通用和复杂环境中具有实用性。
英文摘要：
Equipping embodied agents with commonsense is important for robots to successfully complete complex human instructions in general environments. Recent large language models (LLM) can embed rich semantic knowledge for agents in plan generation of complex tasks, while they lack the information about the realistic world and usually yield infeasible action sequences. In this paper, we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models. Specifically, we first construct a multimodal dataset containing triplets of indoor scenes, instructions and action plans, where we provide the designed prompts and the list of existing objects in the scene for GPT-3.5 to generate a large number of instructions and corresponding planned actions. The generated data is leveraged for grounded plan tuning of pre-trained LLMs. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin, which indicates the practicality of embodied task planning in general and complex environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
