arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.11287cs.CLcs.AI
收藏

基于Q值排序的过程奖励模型

Process Reward Model with Q-Value Rankings

Huazhong University of Science and Technology(华中科技大学);University of the Wisconsin-Madison(威斯康星大学麦迪逊分校)

Wendi Li, Yixuan Li

AI总结:

针对现有过程奖励建模(PRM)方法存在的次优奖励分布及步骤依赖问题,提出过程Q值模型(PQM),通过马尔可夫决策过程框架及对比损失优化Q值排序,实证显示其在多场景下优于分类PRM。

摘要:

过程奖励建模(PRM)对于复杂推理和决策任务至关重要,其中中间步骤的准确性对整体结果有显著影响。现有PRM方法主要被构建为分类问题,采用交叉熵损失独立评估每个步骤的正确性。这种方法可能导致次优的奖励分布,且未能充分解决步骤间的相互依赖关系。为解决这些局限性,我们引入过程Q值模型(PQM),这是一种将PRM重新定义在马尔可夫决策过程语境下的新型框架。PQM基于一种新颖的对比损失函数优化Q值排序,增强模型捕捉序列决策间复杂动态的能力。该方法为过程奖励提供了更精细、理论更扎实的方法论。我们在多种采样策略、语言模型主干和多步推理基准上的广泛实证评估表明,PQM优于基于分类的PRM。我们全面的消融研究突出了对比损失函数的有效性,证实了PQM的实际功效和理论优势。

英文摘要:

Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑