arXiv 2504.152752025-10-23cs.AIcs.LG

停止求和：最小形式信用分配是推理任务过程奖励模型所需的一切

Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tencent（腾讯）；Faculty of Innovation Engineering, Macau University of Science and Technology（澳门科技大学创新工程学院）；DeSci Center of Parallel Intelligence, Obuda University（并行智能德Sci中心，奥布达大学）；Research Center for Chinese Economics and Social Security, University of Chinese Academy of Sciences（中国经济社会安全研究中心，中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）

Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, Fei-Yue Wang

备注：: Accepted by NeurIPS 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出PURE方法，通过最小形式信用分配替代求和形式，缓解过程奖励模型在推理任务中的奖励黑客问题，仅需30%步骤即达可验证奖励方法性能，并在Qwen2.5-Math-7B上实现最优微调。

摘要：

过程奖励模型（PRMs）已被证明在具有挑战性的推理任务上对大型语言模型（LLMs）的测试时扩展有效。然而，PRMs的奖励黑客问题限制了其在强化微调中的成功应用。在本文中，我们确定了PRM诱导的奖励黑客的主要成因：强化学习（RL）中经典的求和形式信用分配，该形式将价值定义为累积的伽马衰减未来奖励，容易诱导LLMs攻击高奖励步骤。为解决此问题，我们提出PURE：过程监督强化学习。PURE的关键创新是最小形式信用分配，将价值函数表述为未来奖励的最小值。该方法通过限制价值函数范围并更合理地分配优势，显著缓解了奖励黑客问题。通过在3个基础模型上的广泛实验，我们表明基于PRM的启用最小形式信用分配的方法在仅30%的步骤内即可达到与可验证奖励方法相当的推理性能。相比之下，经典的求和形式信用分配即使在训练开始时也会导致训练崩溃！此外，当我们仅用10%的可验证奖励补充基于PRM的微调时，我们进一步缓解了奖励黑客问题，并在我们的实验中基于Qwen2.5-Math-7B产生了最佳微调模型，在AMC23上达到82.5%的准确率，在5个基准上平均准确率为53.3%。此外，我们总结了观察到的奖励黑客案例并分析了训练崩溃的原因。我们在https://github.com/CJReinforce/PURE发布代码和模型权重。

英文摘要：

Process reward models (PRMs) have proven effective for test-time scaling of Large Language Models (LLMs) on challenging reasoning tasks. However, reward hacking issues with PRMs limit their successful application in reinforcement fine-tuning. In this paper, we identify the main cause of PRM-induced reward hacking: the canonical summation-form credit assignment in reinforcement learning (RL), which defines the value as cumulative gamma-decayed future rewards, easily induces LLMs to hack steps with high rewards. To address this, we propose PURE: Process sUpervised Reinforcement lEarning. The key innovation of PURE is a min-form credit assignment that formulates the value function as the minimum of future rewards. This method significantly alleviates reward hacking by limiting the value function range and distributing advantages more reasonably. Through extensive experiments on 3 base models, we show that PRM-based approaches enabling min-form credit assignment achieve comparable reasoning performance to verifiable reward-based methods within only 30% steps. In contrast, the canonical sum-form credit assignment collapses training even at the beginning! Additionally, when we supplement PRM-based fine-tuning with just 10% verifiable rewards, we further alleviate reward hacking and produce the best fine-tuned model based on Qwen2.5-Math-7B in our experiments, achieving 82.5% accuracy on AMC23 and 53.3% average accuracy across 5 benchmarks. Moreover, we summarize the observed reward hacking cases and analyze the causes of training collapse. We release our code and model weights at https://github.com/CJReinforce/PURE.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。