arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.07714cs.CL
收藏

StableToolBench:迈向大型语言模型工具学习的稳定大规模基准测试

StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Google(谷歌);The University of Hong Kong(香港大学)

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, Yang Liu

AI总结:

为解决现有工具学习基准中API状态不稳定的问题,本文提出StableToolBench,通过引入虚拟API服务器(含缓存与模拟器)和基于GPT-4的稳定评估系统,实现了大规模且稳定的LLM工具使用能力评估。

摘要:

大型语言模型(LLMs)在近年来取得了显著进展,促使了工具学习的探索,即将LLMs与外部工具相结合以应对各种现实世界的挑战。评估LLMs使用工具的能力需要大规模且稳定的基准测试。然而,以往的研究要么依赖于规模有限的手工在线工具,要么依赖于大规模的真实在线API,但后者饱受API状态不稳定的问题困扰。为了解决这个问题,我们引入了StableToolBench,这是一个从ToolBench演进而来的基准测试,提出了一种虚拟API服务器和稳定的评估系统。虚拟API服务器包含缓存系统和API模拟器,两者互补以缓解API状态的变化。同时,稳定的评估系统设计了可解的通过率和胜率,并使用GPT-4作为自动评估器,以消除评估过程中的随机性。实验结果证明了StableToolBench的稳定性,并进一步讨论了API模拟器、缓存系统和评估器系统的有效性。

英文摘要:

Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑