CRITICTOOL：评估大语言模型在工具调用错误场景下的自我批判能力
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
AI总结：
本研究针对大语言模型工具调用易出错的问题，分析错误类型后构建了专用评估基准CRITICTOOL，通过进化策略生成多样复杂的错误场景，验证了基准有效性并剖析了不同模型的工具反思能力。
摘要：
大语言模型（LLM）利用外部工具的能力使其能够处理日益多样化的任务。然而，随着任务变得更复杂且周期更长，复杂的工具使用过程可能会引发各种意外错误。因此，如何有效处理这类错误，包括识别、诊断并从错误中恢复，已成为推动工具学习发展的关键研究方向。在本研究中，我们首先在多个主流工具评估基准上，对函数调用过程中遇到的错误类型进行了全面分析。在此基础上，我们提出了CRITICTOOL，这是一个专门针对工具学习的综合性批判评估基准。基于一种新颖的进化策略构建数据集，CRITICTOOL包含不同复杂度的多样工具使用错误，更能反映真实场景。我们在CRITICTOOL上开展了大量实验，验证了我们所构建的基准策略的泛化性和有效性。我们还对各类大语言模型的工具反思能力进行了深入分析，为大语言模型工具学习领域提供了新的视角。代码可在https://github.com/Shellorley0513/CriticTool 获取。
英文摘要：
The ability of large language models (LLMs) to utilize external tools has enabled them to tackle an increasingly diverse range of tasks. However, as the tasks become more complex and long-horizon, the intricate tool utilization process may trigger various unexpected errors. Therefore, how to effectively handle such errors, including identifying, diagnosing, and recovering from them, has emerged as a key research direction for advancing tool learning. In this work, we first extensively analyze the types of errors encountered during the function-calling process on several competitive tool evaluation benchmarks. Based on it, we introduce CRITICTOOL, a comprehensive critique evaluation benchmark specialized for tool learning. Building upon a novel evolutionary strategy for dataset construction, CRITICTOOL holds diverse tool-use errors with varying complexities, which better reflects real-world scenarios. We conduct extensive experiments on CRITICTOOL, and validate the generalization and effectiveness of our constructed benchmark strategy. We also provide an in-depth analysis of the tool reflection ability on various LLMs, offering a new perspective on the field of tool learning in LLMs. The code is available at \href{https://github.com/Shellorley0513/CriticTool}{https://github.com/Shellorley0513/CriticTool}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
