arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.02506cs.CL
收藏

ToolHop:一个用于评估大语言模型多跳工具使用能力的查询驱动基准

ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

Fudan University(复旦大学);ByteDance(字节跳动);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心)

Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiecao Chen

AI总结:

针对多跳工具使用评估数据集匮乏的问题,研究提出包含995条查询与3912个工具的查询驱动基准ToolHop,评估14个大语言模型后发现最优模型GPT-4o准确率仅49.04%,为相关方法开发提供了参考。

摘要:

有效评估多跳工具使用能力对于分析大语言模型(LLM)的理解、推理和函数调用能力至关重要。然而,可靠评估数据集的匮乏阻碍了该领域的进展。为解决这一问题,我们提出了ToolHop数据集,它包含995条用户查询和3912个关联工具,专为严格评估多跳工具使用能力而设计。ToolHop通过一种新颖的查询驱动数据构建方法(包括工具创建、文档完善和代码生成),确保了查询的多样性、有意义的相互依赖关系、可本地执行的工具、详细的反馈以及可验证的答案。我们对五个模型家族(即LLaMA3.1、Qwen2.5、Gemini1.5、Claude3.5和GPT)的14个大语言模型进行了评估,发现模型在处理多跳工具使用场景时存在显著挑战。表现最优的模型GPT-4o准确率仅为49.04%,说明该领域仍有很大提升空间。进一步分析揭示了不同模型家族在工具使用策略上的差异,为开发更有效的方法提供了可行的见解。代码和数据可在https://huggingface.co/datasets/bytedance-research/ToolHop获取。

英文摘要:

Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in https://huggingface.co/datasets/bytedance-research/ToolHop.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑