arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.20073cs.CLcs.AIcs.MA
收藏

Collab-Overcooked: 大语言模型作为协作代理的基准测试与评估

Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents

Beijing University of Posts and Telecommunications(北京邮电大学);Li Auto Inc

Haochen Sun, Shuwen Zhang, Lujie Niu, Lei Ren, Hao Xu, Hao Fu, Fangkun Zhao, Caixia Yuan, Xiaojie Wang

AI总结:

本文提出Collab-Overcooked基准测试,评估大语言模型在协作代理中的表现,揭示其在目标解释和协作能力上的优劣。

摘要:

基于大语言模型(LLM)的代理系统在现实应用中已超越传统NLP任务取得了显著进展。本文提出了一种新的LLM多代理系统(LLM-MAS)基准测试,Collab-Overcooked,基于流行的Overcooked-AI游戏,增加了在交互环境中更具应用性和挑战性的任务。Collab-Overcooked以两种新颖的方式扩展了现有基准测试。首先,它提供了一个支持多样化任务和目标的多代理框架,并通过自然语言通信鼓励协作。其次,它引入了一种过程导向的评估指标谱,以评估不同LLM代理的细粒度协作能力,这是先前工作常忽视的维度。我们使用13种流行的LLM进行了广泛的实验,表明虽然LLM在目标解释方面表现出强大的能力,但在主动协作和持续适应方面存在显著不足,这对于高效完成复杂任务至关重要。值得注意的是,我们突显了LLM-MAS的优势和劣势,并提供了在统一且开源的基准上改进和评估LLM-MAS的见解。环境、30个开放性任务以及评估包已公开在https://github.com/YusaeMeow/Collab-Overcooked。

英文摘要:

Large Language Models (LLMs) based agent systems have made great strides in real-world applications beyond traditional NLP tasks. This paper proposes a new LLM-based Multi-Agent System (LLM-MAS) benchmark, Collab-Overcooked, built on the popular Overcooked-AI game with more applicable and challenging tasks in interactive environments. Collab-Overcooked extends existing benchmarks in two novel ways. First, it provides a multi-agent framework supporting diverse tasks and objectives and encourages collaboration through natural language communication. Second, it introduces a spectrum of process-oriented evaluation metrics to assess the fine-grained collaboration capabilities of different LLM agents, a dimension often overlooked in prior work. We conduct extensive experiments with 13 popular LLMs and show that, while the LLMs exhibit a strong ability in goal interpretation, there are significant shortcomings in active collaboration and continuous adaptation, which are critical for efficiently fulfilling complex tasks. Notably, we highlight the strengths and weaknesses of LLM-MAS and provide insights for improving and evaluating LLM-MAS on a unified and open-source benchmark. The environments, 30 open-ended tasks, and the evaluation package are publicly available at https://github.com/YusaeMeow/Collab-Overcooked.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑