arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.26535cs.AI
收藏

PAPO:通过解耦优势归一化稳定规则整合训练

PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization

University of Science and Technology of China(中国科学技术大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);University of Oxford(牛津大学);Wuhan University(武汉大学)

Zelin Tan, Zhouliang Yu, Bohan Lin, Zijie Geng, Hejia Geng, Yudong Zhang, Mulei Zhang, Yang Chen, Shuyue Hu, Zhenfei Yin, Chen Zhang, Lei Bai

AI总结:

PAPO通过解耦优势归一化整合过程级评估,解决现有奖励设计的两个局限:Outcome Reward Models(ORM)仅评估最终答案正确性,而Process Reward Models(PRM)虽监督更丰富但易导致奖励黑客。

摘要:

我们提出Process-Aware Policy Optimization(PAPO),通过解耦优势归一化将过程级评估整合到Group Relative Policy Optimization(GRPO)中,以解决现有奖励设计的两个局限。Outcome Reward Models(ORM)仅评估最终答案的正确性,无论推理质量如何,且当组内所有回答均正确时,优势信号会逐渐消失。Process Reward Models(PRM)提供更丰富的监督,但直接使用PRM分数会导致奖励黑客,即模型通过冗长性来提高分数,而准确性却崩溃。PAPO通过将优势分为一个来自ORM的成分Aout(基于ORM归一化所有响应)和一个来自基于规则的PRM的成分Aproc(仅在正确响应中归一化)来解决这两个问题。这种解耦设计确保Aout将训练锚定在正确性上,而Aproc则区分推理质量而不扭曲结果信号。在多个模型规模和六个基准测试中,实验表明PAPO在OlympiadBench上达到51.3% vs. 46.3%,并在ORM plateau和下降时持续改进。

英文摘要:

We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through decoupled advantage normalization, to address two limitations of existing reward designs. Outcome reward models (ORM) evaluate only final-answer correctness, treating all correct responses identically regardless of reasoning quality, and gradually lose the advantage signal as groups become uniformly correct. Process reward models (PRM) offer richer supervision, but directly using PRM scores causes reward hacking, where models exploit verbosity to inflate scores while accuracy collapses. PAPO resolves both by composing the advantage from an outcome component Aout, derived from ORM and normalized over all responses, and a process component Aproc, derived from a rubric-based PRM and normalized exclusively among correct responses. This decoupled design ensures that Aout anchors training on correctness while Aproc differentiates reasoning quality without distorting the outcome signal. Experiments across multiple model scales and six benchmarks demonstrate that PAPO consistently outperforms ORM, reaching 51.3% vs.\ 46.3% on OlympiadBench while continuing to improve as ORM plateaus and declines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑