通过对比互信息实现高效的进程奖励建模
Efficient Process Reward Modeling via Contrastive Mutual Information
AI总结：
本文提出CPMI方法，通过模型内部概率推断步骤监督，减少标注计算负担，提升过程级评估和数学推理准确性。
摘要：
近期研究致力于通过进程奖励模型（PRMs）等验证器模型验证链式思维（CoT）轨迹的中间推理步骤。然而，训练PRM通常需要人工标注员为每个推理步骤分配奖励分数，这既昂贵又耗时。现有自动化方法如蒙特卡洛（MC）估计也因重复LLM rollout而消耗大量计算资源。为克服这些限制，我们提出对比点互信息（CPMI），一种新颖的自动奖励标注方法，利用模型内部概率推断步骤级监督，显著减少标注数据集的计算负担。CPMI量化推理步骤相对于硬负样本增加的互信息，此对比信号作为步骤对最终解决方案的贡献代理，产生可靠奖励。实验结果表明，基于CPMI的标注将数据集构建时间减少84%，token生成减少98%，同时在过程级评估和数学推理基准上实现更高准确性。
英文摘要：
Recent research has devoted considerable effort to verifying the intermediate reasoning steps of chain-of-thought (CoT) trajectories using process reward models (PRMs) and other verifier models. However, training a PRM typically requires human annotators to assign reward scores to each reasoning step, which is both costly and time-consuming. Existing automated approaches, such as Monte Carlo (MC) estimation, also demand substantial computational resources due to repeated LLM rollouts. To overcome these limitations, we propose contrastive pointwise mutual information (CPMI), a novel automatic reward labeling method that leverages the model's internal probability to infer step-level supervision while significantly reducing the computational burden of annotating dataset. CPMI quantifies how much a reasoning step increases the mutual information between the step and the correct target answer relative to hard-negative alternatives. This contrastive signal serves as a proxy for the step's contribution to the final solution and yields a reliable reward. The experimental results show that CPMI-based labeling reduces dataset construction time by 84% and token generation by 98% compared to MC estimation, while achieving higher accuracy on process-level evaluations and mathematical reasoning benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
