arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.07166cs.CLcs.AIcs.LGcs.RO
收藏

具身智能体接口：面向具身决策的LLM基准测试

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

Stanford University（斯坦福大学）；Northwestern University（西北大学）；Amazon（亚马逊公司）；MIT（麻省理工学院）
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-… 展开作者

Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu

备注：
Accepted for oral presentation at NeurIPS 2024 in the Datasets and Benchmarks track. Final Camera version
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出具身智能体接口，统一任务、模块和细粒度指标，系统评估LLMs在具身决策中的能力，定位错误类型，指导有效利用。

摘要：

我们旨在评估大型语言模型（LLMs）在具身决策中的表现。尽管已有大量工作利用LLMs在具身环境中进行决策，但由于这些工作通常应用于不同领域、服务于不同目的，并基于不同的输入和输出构建，我们仍缺乏对其性能的系统性理解。此外，现有评估往往仅依赖最终成功率，难以定位LLMs缺失的能力及问题所在，这阻碍了具身智能体有效且有选择地利用LLMs。为解决这些局限，我们提出了一种通用接口（具身智能体接口），支持对各类任务及基于LLM的模块的输入输出规范进行形式化。具体而言，该接口使我们能够统一：1）涉及状态目标和时间延展目标的广泛具身决策任务；2）四种常用的基于LLM的决策模块：目标解释、子目标分解、动作排序和状态转移建模；3）一组细粒度指标，将评估细分为多种错误类型，如幻觉错误、可供性错误、各类规划错误等。总体而言，我们的基准对LLMs在不同子任务上的性能进行了全面评估，指出了LLM驱动的具身AI系统的优缺点，并为在具身决策中有效且有选择地使用LLMs提供了见解。

英文摘要：

We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑