ToolSandbox：面向大语言模型工具使用能力的有状态、对话式、交互式评估基准
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
AI总结：
针对LLM工具使用能力评估缺乏全面性的问题，提出ToolSandbox基准，包含有状态工具执行、工具间隐式依赖、在线策略对话评估等特性，揭示开源与专有模型性能差距及复杂任务挑战，为工具使用LLM能力评估提供新方法。
摘要：
近期大语言模型（LLMs）的进步激发了研究人员对工具辅助LLMs解决现实挑战的兴趣，这需要对工具使用能力进行全面评估。此前工作或聚焦无状态网络服务（RESTful API）评估、基于单轮用户提示，或基于离线策略对话轨迹；而ToolSandbox包含有状态工具执行、工具间隐式状态依赖、支持在线策略对话评估的内置用户模拟器，以及对任意轨迹中间和最终里程碑的动态评估策略。我们发现开源与专有模型存在显著性能差距，ToolSandbox定义的状态依赖、规范化、信息不足等复杂任务对最先进的LLMs也具挑战性，为工具使用LLM能力提供新见解。ToolSandbox评估框架已发布于https://github.com/apple/ToolSandbox
英文摘要：
Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
