Reward Reasoning Model（奖励推理模型）
Reward Reasoning Model
AI总结：
针对测试时计算量难以有效用于提升奖励模型性能的问题，研究提出Reward Reasoning Models，通过强化学习框架培育自我进化的思维链奖励推理能力，可自适应利用测试时计算，在多领域奖励建模基准上取得更优性能。
摘要：
奖励模型在引导大语言模型生成符合人类预期的输出方面发挥着关键作用。然而，如何有效利用测试时计算量来提升奖励模型性能仍是一个待解决的公开挑战。在本研究中，我们提出了Reward Reasoning Models（RRMs，奖励推理模型），该模型专门设计为在生成最终奖励前执行审慎的推理过程。通过思维链推理，RRMs能够在合适的奖励并非一目了然的复杂查询场景下，调用额外的测试时计算量。为开发RRMs，我们搭建了一个强化学习框架，该框架无需显式推理轨迹作为训练数据，即可培育出自我进化的奖励推理能力。实验结果表明，RRMs在多个不同领域的奖励建模基准测试中均取得了更优的性能。值得注意的是，我们证实RRMs能够自适应地利用测试时计算量，进一步提升奖励准确率。预训练的奖励推理模型已发布于https://huggingface.co/Reward-Reasoning。
英文摘要：
Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained reward reasoning models are available at https://huggingface.co/Reward-Reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
