arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01511cs.CLcs.LG
收藏

交替强化学习用于非可验证大语言模型后训练的评分标准建模

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training

Emory University(埃默里大学);Purdue University(普渡大学);Rutgers University(罗格斯大学);Georgia Institute of Technology(佐治亚理工学院);University at Albany(阿尔巴尼大学)

Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, Haoyu Wang

AI总结:

Rubric-ARM通过交替强化学习优化评分标准生成与评判,提升非可验证领域大语言模型后训练的响应质量评估与策略对齐性能。

摘要:

标准奖励模型通常预测标量分数,难以捕捉非可验证领域(如创意写作或开放式指令跟随)中响应质量的多维特性。为解决这一限制,我们提出了Rubric-ARM框架,该框架通过强化学习从偏好反馈中联合优化评分标准生成器和评判器。不同于现有方法依赖静态评分标准或分离的训练流程,我们的方法将评分标准生成视为一个潜在动作,通过最大化判断准确性来学习。我们引入了一种交替优化策略以缓解同时更新带来的非平稳性,并提供了理论分析,证明这种调度方式在训练过程中如何减少梯度方差。大量实验表明,Rubric-ARM在多个基准测试中实现了优于基线的性能,并在离线和在线强化学习设置中显著提升了下游策略对齐效果。

英文摘要:

Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑