R-Align: 通过以推理为中心的元判断增强生成奖励模型
R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging
AI总结:
R-Align通过引入黄金判断和明确监督推理对齐,提高生成奖励模型的推理一致性,从而增强RLHF的性能。
摘要:
基于人类反馈的强化学习(RLHF)仍然对于在主观领域对大型语言模型(LLMs)进行对齐是必不可少的。为了提高鲁棒性,近期的研究转向生成奖励模型(GenRMs),这些模型在预测偏好之前会生成推理过程。然而,在GenRM的训练和评估中,实践仍然只依赖于结果标签,导致推理质量未被检查。我们证明了推理一致性——生成奖励模型的偏好决策与参考决策推理之间的一致性——高度预测下游RLHF结果,而不仅仅是标准标签准确性。具体来说,我们重新利用现有的奖励模型基准来计算虚假正确性(S-Corr)——正确标签决策中推理与黄金判断不一致的分数比例。我们的实证评估揭示了即使对于竞争性的GenRMs,S-Corr仍然相当显著,且更高的S-Corr与在优化过程中策略退化有关。为了提高一致性,我们提出了以推理为中心的对齐,即R-Align,它通过加入黄金判断来增强训练,并明确监督推理对齐。R-Align在RM基准上减少了S-Corr,并在STEM、编程、遵循指令和一般任务中产生了持续的演员性能提升。
英文摘要:
Reinforcement Learning from Human Feedback (RLHF) remains indispensable for aligning large language models (LLMs) in subjective domains. To enhance robustness, recent work shifts toward Generative Reward Models (GenRMs) that generate rationales before predicting preferences. Yet in GenRM training and evaluation, practice remains outcome-label-only, leaving reasoning quality unchecked. We show that reasoning fidelity-the consistency between a GenRM's preference decision and reference decision rationales-is highly predictive of downstream RLHF outcomes, beyond standard label accuracy. Specifically, we repurpose existing reward-model benchmarks to compute Spurious Correctness (S-Corr)-the fraction of label-correct decisions with rationales misaligned with golden judgments. Our empirical evaluation reveals substantial S-Corr even for competitive GenRMs, and higher S-Corr is associated with policy degeneration under optimization. To improve fidelity, we propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment. R-Align reduces S-Corr on RM benchmarks and yields consistent gains in actor performance across STEM, coding, instruction following, and general tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。