BattleAgentBench：一个用于评估多智能体系统中语言模型合作与竞争能力的基准
BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systems
AI总结：
针对现有基准缺乏对多智能体协作与竞争场景细粒度评估的问题，提出BattleAgentBench基准，通过三个难度级别的七个子阶段全面评估语言模型在导航、任务执行及多智能体协作竞争方面的能力，实验表明API模型在简单任务表现优异但困难任务仍需大幅改进。
摘要：
大型语言模型（LLMs）正变得越来越强大，能够处理复杂任务，例如构建单智能体和多智能体系统。与单智能体相比，多智能体系统对语言模型的协作能力有更高要求。目前已提出许多基准来评估其协作能力。然而，这些基准缺乏对LLM协作能力的细粒度评估。此外，现有的工作忽略了多智能体协作与竞争场景。为了解决这两个问题，我们提出了一个名为BattleAgentBench的基准，它定义了三个不同难度级别的七个子阶段，并在单智能体场景导航能力、成对智能体任务执行能力以及多智能体协作与竞争能力方面对语言模型进行了细粒度评估。我们对领先的四个闭源模型和七个开源模型进行了广泛评估。实验结果表明，基于API的模型在简单任务上表现优异，但开源小模型在简单任务上表现挣扎。对于需要协作和竞争能力的困难任务，尽管基于API的模型展示了一定的协作能力，但仍有巨大的改进空间。
英文摘要：
Large Language Models (LLMs) are becoming increasingly powerful and capable of handling complex tasks, e.g., building single agents and multi-agent systems. Compared to single agents, multi-agent systems have higher requirements for the collaboration capabilities of language models. Many benchmarks are proposed to evaluate their collaborative abilities. However, these benchmarks lack fine-grained evaluations of LLM collaborative capabilities. Additionally, multi-agent collaborative and competitive scenarios are ignored in existing works. To address these two problems, we propose a benchmark, called BattleAgentBench, which defines seven sub-stages of three varying difficulty levels and conducts a fine-grained evaluation of language models in terms of single-agent scenario navigation capabilities, paired-agent task execution abilities, and multi-agent collaboration and competition capabilities. We conducted extensive evaluations on leading four closed-source and seven open-source models. Experimental results indicate that API-based models perform excellently on simple tasks but open-source small models struggle with simple tasks. Regarding difficult tasks that require collaborative and competitive abilities, although API-based models have demonstrated some collaborative capabilities, there is still enormous room for improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
