arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.13622cs.AI
收藏

面向数学推理的由粗到细过程奖励建模

Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning

Renmin University of China（中国人民大学）；Gaoling School of Artificial Intelligence（甘肃学校人工智能学院）；University of Toronto（多伦多大学）

Yulan Hu, Sheng Ouyang, Jinman Zhao, Yong Liu

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大语言模型数学推理步骤中的冗余问题，本文提出由粗到细的过程奖励建模方法CFPRM，通过多粒度窗口合并与细化推理步骤，在保留细粒度知识的同时缓解冗余，并在多个数据集和损失准则上验证了其有效性。

摘要：

过程奖励模型（PRM）在数学推理任务中发挥着关键作用，需要高质量的监督过程数据。然而，我们观察到，由大语言模型（LLMs）生成的推理步骤往往无法呈现严格的增量信息，导致冗余，从而阻碍有效推理。为解决这一问题，我们提出了CFPRM，一种简单而有效的由粗到细策略。我们的方法不专注于检测冗余步骤，而是首先建立一个粗粒度窗口，将相邻推理步骤合并为统一、整体的步骤。随后逐步缩小窗口大小以提取细粒度推理步骤，从而能够在多个粒度上收集训练数据。通过利用这种层次化细化过程，CFPRM在保留关键细粒度知识的同时缓解了冗余。在两个推理数据集上、跨三种损失准则的广泛实验验证了CFPRM的有效性和通用性。

英文摘要：

The Process Reward Model (PRM) plays a crucial role in mathematical reasoning tasks, requiring high-quality supervised process data. However, we observe that reasoning steps generated by Large Language Models (LLMs) often fail to exhibit strictly incremental information, leading to redundancy that can hinder effective reasoning. To address this issue, we propose CFPRM, a simple yet effective coarse-to-fine strategy. Instead of focusing on the detection of redundant steps, our approach first establishes a coarse-grained window to merge adjacent reasoning steps into unified, holistic steps. The window size is then progressively reduced to extract fine-grained reasoning steps, enabling data collection at multiple granularities for training. By leveraging this hierarchical refinement process, CFPRM mitigates redundancy while preserving essential fine-grained knowledge. Extensive experiments on two reasoning datasets across three loss criteria validate the CFPRM's effectiveness and versatility.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑