结果准确性不够：对奖励模型推理过程的对齐
Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models
AI总结：
本文提出推理一致性度量，结合结果准确性改进生成奖励模型训练，提升RLHF性能并减少欺骗性对齐问题。
摘要：
生成奖励模型（GenRMs）和大语言模型作为判断者表现出欺骗性对齐，因为它们在训练和评估时优先考虑结果准确性，这削弱了它们在强化学习人类反馈（RLHF）中的泛化能力。我们引入了推理一致性，这是一个细粒度的度量标准，用于量化模型的推理过程与人类判断之间的对齐程度。我们对前沿模型的评估表明，推理一致性能够有效区分最先进的模型，并检测欺骗性对齐，而结果准确性在两个方面都显得不足。为缓解这一差距，我们引入了一种混合信号，将推理一致性与结果准确性结合用于GenRM训练。我们的训练方法在RM-Bench（87.1%）和JudgeBench（82%）上实现了最先进的性能，比仅依赖结果的基线高出平均5%。使用RM进行RLHF时，我们的方法有效提高了性能，如在Arena Hard v2上得到验证，特别是在创造性写作任务中实现了7%的提升。进一步分析确认，我们的方法摆脱了欺骗性对齐陷阱，有效逆转了仅依赖结果训练中观察到的推理一致性下降。
英文摘要：
Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
