基于记忆增强大型语言模型的开放式可指导具身智能体
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
AI总结：
本文提出HELPER具身智能体，通过外部记忆和检索增强LLM提示解析自由形式对话为动作程序，在TEACh基准测试中创下新纪录。
摘要：
预训练且冻结的大型语言模型（LLMs）可以通过适当的少样本示例提示，有效地将简单的场景重排指令映射为机器人视觉运动功能上的程序。然而，为了解析开放域自然语言并适应用户在提示工程阶段未知的特有流程，固定提示存在不足。在本文中，我们引入了HELPER，这是一个配备了语言-程序对外部记忆的具身智能体，它通过检索增强的LLM提示将自由形式的人机对话解析为动作程序：基于当前对话、指令、纠正或VLM描述检索相关记忆，并将其作为LLM查询的上下文提示示例。该记忆在部署期间会进行扩展，以包含用户语言和动作计划的配对，从而协助未来的推理并使其个性化以适应用户的语言和日常惯例。HELPER在TEACh基准测试中，无论是基于对话历史的执行（EDH）还是基于对话的轨迹（TfD），均创下了新的最优记录，其中TfD相较于先前的最优记录提升了1.7倍。我们的模型、代码和视频结果可在项目网站找到：https://helper-agent-llm.github.io。
英文摘要：
Pre-trained and frozen large language models (LLMs) can effectively map simple scene rearrangement instructions to programs over a robot's visuomotor functions through appropriate few-shot example prompting. To parse open-domain natural language and adapt to a user's idiosyncratic procedures, not known during prompt engineering time, fixed prompts fall short. In this paper, we introduce HELPER, an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting: relevant memories are retrieved based on the current dialogue, instruction, correction, or VLM description, and used as in-context prompt examples for LLM querying. The memory is expanded during deployment to include pairs of user's language and action plans, to assist future inferences and personalize them to the user's language and routines. HELPER sets a new state-of-the-art in the TEACh benchmark in both Execution from Dialog History (EDH) and Trajectory from Dialogue (TfD), with a 1.7x improvement over the previous state-of-the-art for TfD. Our models, code, and video results can be found in our project's website: https://helper-agent-llm.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
