arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07484cs.AIcs.CLcs.LG
收藏

ConsistRM:通过一致性感知自训练改进生成奖励模型

ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training

Baidu Inc.(百度公司)

Yu Liang, Liangxin Liu, Longzheng Wang, Yan Wang, Yueyang Zhang, Long Xia, Zhiyuan Sun, Daiting Shi

AI总结:

本文提出ConsistRM框架,通过一致性感知答案奖励和批评奖励提升生成奖励模型的稳定性与一致性,实验表明其在五个基准数据集上优于传统强化微调方法。

摘要:

生成奖励模型(GRMs)通过提供更大的表示能力和灵活性,成为对齐大语言模型(LLMs)与人类偏好的一种有前景的方法。然而,GRMs面临两个主要挑战:依赖昂贵的人工标注数据限制了可扩展性,而自训练方法常因不稳定性和奖励黑客问题而受损。为解决这些问题,我们提出了ConsistRM,一种无需人工标注的自训练框架。ConsistRM整合了一致性感知答案奖励,通过时间一致性生成可靠的伪标签,从而提供更稳定的模型优化。此外,引入了一致性感知批评奖励,以评估多批评之间的语义一致性,并分配细粒度和差异化的奖励。在五个基准数据集上对四个基础模型的实验表明,ConsistRM在传统强化微调(RFT)上平均提升1.5%。进一步分析显示,ConsistRM增强了输出一致性并减轻了输入顺序导致的位置偏差,突显了一致性感知奖励在改进GRMs中的有效性。我们的实现可在https://github.com/yuliangCarmelo/ConsistRM获取。

英文摘要:

Generative reward models (GRMs) have emerged as a promising approach for aligning Large Language Models (LLMs) with human preferences by offering greater representational capacity and flexibility than traditional scalar reward models. However, GRMs face two major challenges: reliance on costly human-annotated data restricts scalability, and self-training approaches often suffer from instability and vulnerability to reward hacking. To address these issues, we propose ConsistRM, a self-training framework that enables effective and stable GRM training without human annotations. ConsistRM incorporates the Consistency-Aware Answer Reward, which produces reliable pseudo-labels with temporal consistency, thereby providing more stable model optimization. Moreover, the Consistency-Aware Critique Reward is introduced to assess semantic consistency across multiple critiques and allocates fine-grained and differentiated rewards. Experiments on five benchmark datasets across four base models demonstrate that ConsistRM outperforms vanilla Reinforcement Fine-Tuning (RFT) by an average of 1.5%. Further analysis shows that ConsistRM enhances output consistency and mitigates position bias caused by input order, highlighting the effectiveness of consistency-aware rewards in improving GRMs. Our implementation is available at https://github.com/yuliangCarmelo/ConsistRM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑