ReflectRM: 通过统一判断框架内的自我反思提升生成奖励模型
ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
AI总结：
ReflectRM通过自我反思评估分析质量，提升偏好建模，实验表明其在四个基准测试中平均准确率提升3.7%，并有效缓解位置偏差。
摘要：
奖励模型（RMs）是强化学习从人类反馈（RLHF）流程中的关键组件，直接决定大型语言模型（LLMs）的对齐质量。最近，生成奖励模型（GRMs）作为更优范式出现，相较于传统标量RMs，提供更高的可解释性和更强的泛化能力。然而，现有GRM方法主要关注结果级监督，忽视了分析过程质量，限制了其潜力。为此，我们提出ReflectRM，一种新型GRM，利用自我反思评估分析质量并增强偏好建模。ReflectRM在统一生成框架下联合建模响应偏好和分析偏好。推理时，利用其自我反思能力识别最可靠的分析，从而得出最终偏好预测。在四个基准测试中，实验表明ReflectRM性能一致提升，平均准确率提升+3.7（Qwen3-4B）。进一步实验证实响应偏好和分析偏好相互促进。值得注意的是，ReflectRM显著缓解位置偏差，相比领先GRMs提升+10.2，确立了更稳定的评估器。我们的代码可在https://github.com/yuliangCarmelo/ReflectRM获取。
英文摘要：
Reward Models (RMs) are critical components in the Reinforcement Learning from Human Feedback (RLHF) pipeline, directly determining the alignment quality of Large Language Models (LLMs). Recently, Generative Reward Models (GRMs) have emerged as a superior paradigm, offering higher interpretability and stronger generalization than traditional scalar RMs. However, existing methods for GRMs focus primarily on outcome-level supervision, neglecting analytical process quality, which constrains their potential. To address this, we propose ReflectRM, a novel GRM that leverages self-reflection to assess analytical quality and enhance preference modeling. ReflectRM is trained under a unified generative framework for joint modeling of response preference and analysis preference. During inference, we use its self-reflection capability to identify the most reliable analysis, from which the final preference prediction is derived. Experiments across four benchmarks show that ReflectRM consistently improves performance, achieving an average accuracy gain of +3.7 on Qwen3-4B. Further experiments confirm that response preference and analysis preference are mutually reinforcing. Notably, ReflectRM substantially mitigates positional bias, yielding +10.2 improvement compared with leading GRMs and establishing itself as a more stable evaluator. Our code is available at https://github.com/yuliangCarmelo/ReflectRM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
