arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.03124cs.CLcs.AIcs.LG
收藏

PRMBench:面向过程级奖励模型的细粒度且具挑战性的基准

PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models

Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Soochow University(苏州大学);Stony Brook University(石溪大学);The Chinese University of Hong Kong(香港中文大学)

Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, Yu Cheng

AI总结:

针对现有基准仅关注步骤正确性的不足,提出PRMBench基准,包含6,216个问题和83,456个步骤级标签,从简洁性、合理性和敏感性多维度评估过程级奖励模型的细粒度错误检测能力,实验揭示了当前模型的显著弱点。

摘要:

过程级奖励模型(PRMs)对于复杂推理和决策任务至关重要,其中每个中间步骤在推理过程中都发挥着重要作用。由于语言模型在推理过程中容易出现各种类型的错误,因此要求PRMs具备检测现实场景中各种隐式错误类型的细微能力。然而,当前的基准主要关注步骤正确性,未能系统地评估PRMs的性能。为了弥补这一差距,我们引入了PRMBench,这是一个专门设计用于评估PRMs细粒度错误检测能力的过程级基准。PRMBench包含6,216个精心设计的问题和83,456个步骤级标签,从多个维度评估模型,包括简洁性、合理性和敏感性。在我们对15个模型的实验中,涵盖开源PRMs和被提示作为评论家模型的闭源大语言模型,我们发现了当前PRMs存在的显著弱点。这些发现强调了过程级评估固有的挑战,并突出了未来研究的关键方向。我们希望PRMBench能成为推进PRM评估与发展研究的稳健基准。

英文摘要:

Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since language models are prone to various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs' performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 15 models, spanning both open-source PRMs and closed-source large language models prompted as critic models, we uncover significant weaknesses in current PRMs. These findings underscore the challenges inherent in process-level evaluation and highlight key directions for future research. We hope PRMBench can be a robust bench for advancing research on PRM evaluation and development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑