arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.16499cs.CL
收藏

LLMArena:评估大型语言模型在动态多智能体环境中的能力

LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments

Beijing University of Posts and Telecommunications(北京邮电大学);Paradigm Inc.(第四范式公司);Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, Lijie Wen

AI总结:

本文提出LLMArena框架,通过七个游戏环境和Trueskill评分在多智能体动态场景中评估LLM的空间推理、规划、协作等能力,实验表明LLM在对手建模与团队协作方面仍存在明显不足。

摘要:

大型语言模型(LLMs)的最新进展揭示了它们实现具有人类水平智能的自主智能体的潜力。然而,现有的用于评估LLM智能体的基准要么使用静态数据集,可能导致数据泄漏,要么只关注单智能体场景,忽视了多智能体交互的复杂性。目前缺乏一个在多智能体、动态环境中评估LLM智能体多样化能力的基准。为此,我们提出了LLMArena,这是一个新颖且易于扩展的框架,用于在多智能体动态环境中评估LLM的多样化能力。LLMArena包含七个不同的游戏环境,采用Trueskill评分来评估LLM智能体的关键能力,包括空间推理、战略规划、数值推理、风险评估、沟通、对手建模和团队协作。我们在不同规模和类型的LLM上进行了广泛的实验和人类评估,结果表明LLM在发展成为完全自主智能体的道路上仍有很长的路要走,尤其是在对手建模和团队协作方面。我们希望LLMArena能够指导未来的研究,以增强LLM的这些能力,最终在动态、多智能体环境中实现更复杂和实用的应用。代码和数据将公开。

英文摘要:

Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMArena, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMArena encompasses seven distinct gaming environments, employing Trueskill scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMArena could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings. The code and data will be available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑