arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.12294cs.AIcs.SE
收藏

ToolPRMBench: 评估和推进用于工具使用智能体的过程奖励模型

ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

Arizona State University(亚利桑那州立大学);Intuit AI Research(Intuit人工智能研究)

Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo

AI总结:

ToolPRMBench通过构建大规模基准评估工具使用智能体的过程奖励模型,揭示PRM有效性差异并展示专用PRMs的潜力。

摘要:

基于奖励的搜索方法在通过有效引导采样和探索复杂动作空间来增强工具使用智能体方面展现出了强大的潜力。作为核心设计,这些搜索方法利用过程奖励模型(PRMs)提供步骤级奖励,使监控更加精细。然而,在工具使用场景中缺乏系统且可靠的PRM评估基准。在本文中,我们引入了ToolPRMBench,一个大规模基准,专门用于评估工具使用智能体的PRMs。ToolPRMBench建立在几个代表性的工具使用基准之上,并将智能体轨迹转换为步骤级测试用例。每个用例包含交互历史、正确动作、一个合理但错误的替代动作以及相关工具元数据。我们分别利用离线采样来隔离局部单步错误,并利用在线采样来捕捉从完整智能体运行中捕获的现实多步失败。我们提出一个多LLM验证流水线以减少标签噪声并确保数据质量。我们在ToolPRMBench上对大型语言模型、通用PRMs和工具专用PRMs进行了广泛的实验。结果揭示了PRM有效性之间的明显差异,并突显了专用PRMs在工具使用方面的潜力。代码和数据将在https://github.com/David-Li0406/ToolPRMBench上发布。

英文摘要:

Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑