arXiv 2401.171672024-06-03cs.CL

规划、创建、使用：在真实世界复杂场景中全面基准测试LLMs的工具利用能力

Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios

Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Huawei Technologies Co., Ltd（华为技术有限公司）；The University of Hong Kong（香港大学）；Peng Cheng Laboratory, Shenzhen（鹏城实验室）

Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, Qun Liu

备注：: Accepted by ACL2024 Findings

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出UltraTool基准测试，全面评估LLMs在真实复杂场景下从规划、创建到使用工具的全流程能力，通过自然语言独立评估规划并消除预定义工具集限制，为工具利用提供新视角。

摘要：

近期将大型语言模型（LLMs）作为工具智能体应用于真实世界场景的趋势，凸显了对其能力进行全面评估的必要性，尤其是在涉及规划、创建和使用工具的复杂场景中。然而，现有的基准测试通常侧重于简单的合成查询，无法反映真实世界的复杂性，因此在评估工具利用方面提供的视角有限。为了解决这一问题，我们提出了UltraTool，这是一个旨在改进和评估LLMs在真实场景中工具利用能力的新型基准测试。UltraTool专注于使用工具的整个过程——从规划、创建到将其应用于复杂任务。它强调真实世界的复杂性，要求进行准确的多步规划以有效解决问题。UltraTool的一个关键特征是其使用自然语言对规划进行独立评估，该规划发生在工具使用之前，并通过规划出中间步骤来简化任务求解。因此，与以往工作不同，它消除了预定义工具集的限制。通过对各种LLMs进行广泛实验，我们为评估LLMs在工具利用方面的能力提供了新见解，从而为这一快速发展的领域贡献了新视角。该基准测试已在 https://github.com/JoeYing1019/UltraTool 公开提供。

英文摘要：

The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools. However, existing benchmarks typically focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. To address this issue, we present UltraTool, a novel benchmark designed to improve and evaluate LLMs' ability in tool utilization within real-world scenarios. UltraTool focuses on the entire process of using tools - from planning and creating to applying them in complex tasks. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage and simplifies the task solving by mapping out the intermediate steps. Thus, unlike previous work, it eliminates the restriction of pre-defined toolset. Through extensive experiments on various LLMs, we offer novel insights into the evaluation of capabilities of LLMs in tool utilization, thereby contributing a fresh perspective to this rapidly evolving field. The benchmark is publicly available at https://github.com/JoeYing1019/UltraTool.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。