arXiv 2501.128512025-11-20cs.CL

ACEBench: 工具使用场景中谁胜出？

ACEBench: Who Wins the Match Point in Tool Usage?

University of Science and Technology of China（中国科学技术大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Shanghai Jiao Tong University（上海交通大学）

Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoq… 展开作者

Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, Wu Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

ACEBench是一个用于评估大型语言模型工具使用能力的综合基准测试，通过三种类型的数据评估场景，深入分析LLMs在不同情境下的工具使用表现。

摘要：

大型语言模型（LLMs）在决策和推理方面展现出了显著的潜力，尤其是在与各种工具结合使用以有效解决复杂问题时。然而，现有评估LLMs工具使用能力的基准测试面临几个限制：（1）评估场景有限，通常缺乏对真实多轮对话情境的评估；（2）评估维度狭窄，对LLMs如何使用工具的详细评估不足；（3）依赖LLMs或真实API执行进行评估，这会引入显著的开销。为了解决这些挑战，我们引入了ACEBench，一个全面的基准测试，用于评估LLMs的工具使用能力。ACEBench根据评估方法将数据分为三种主要类型：正常、特殊和代理。'正常'评估基本场景中的工具使用；'特殊'评估在模糊或不完整指令情况下的工具使用；'代理'通过多代理互动来评估工具使用，以模拟真实世界中的多轮对话。我们使用ACEBench进行了广泛的实验，深入分析了各种LLMs，并提供了对不同数据类型中错误原因的更细致考察。

英文摘要：

Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems. However, existing benchmarks for evaluating LLMs' tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead. To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs. ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. "Normal" evaluates tool usage in basic scenarios; "Special" evaluates tool usage in situations with ambiguous or incomplete instructions; "Agent" evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues. We conducted extensive experiments using ACEBench, analyzing various LLMs in-depth and providing a more granular examination of error causes across different data types.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。