arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.23923cs.CLcs.AI
收藏

Act-Adaptive Margin: 动态校准奖励模型以应对主观模糊性

Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);University of New South Wales(新南威尔士大学);Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团)

Feiteng Fang, Dingwei Chen, Xiang Huang, Ting-En Lin, Yuchuan Wu, Xiong Liu, Xinge Ye, Ziqiang Liu, Haonan Zhang, Liang Zhu, Hamid Alinejad-Rokny, Min Yang, Yongbin Li

AI总结:

本文提出AAM方法,通过动态校准奖励模型偏好边际,提升主观任务中的奖励建模性能,实验显示在角色扮演任务中性能提升显著。

摘要:

目前,大多数强化学习任务集中在数学和编程等领域,这些领域的验证相对简单。然而,在主观任务如角色扮演中,对齐技术难以取得进展,主要原因是使用Bradley-Terry模型进行主观奖励建模时,在处理模糊偏好方面面临重大挑战。为了改进主观任务中的奖励建模,本文提出了AAM(Act-Adaptive Margin),通过利用模型内部参数知识动态校准偏好边际来增强奖励建模。我们设计了AAM的两个版本,能够高效地生成上下文合适的目的偏好差距,而无需额外的人类标注。这种方法从根本上改进了奖励模型处理主观奖励的方式,通过更好地整合生成理解与偏好评分。为了验证AAM在主观奖励建模中的有效性,我们在RewardBench、JudgeBench和具有挑战性的角色扮演任务上进行了评估。结果表明,AAM显著提高了主观奖励建模性能,在一般任务中使Bradley-Terry奖励模型提升了2.95%,在主观角色扮演任务中提升了4.85%。此外,使用AAM训练的奖励模型能够帮助下游对齐任务取得更好结果。我们的测试结果表明,将由AAM增强的RM生成的奖励应用于偏好学习技术(如GRPO)在CharacterEval和Charm上取得了最先进的结果。代码和数据集可在https://github.com/calubkk/AAM上获得。

英文摘要:

Currently, most reinforcement learning tasks focus on domains like mathematics and programming, where verification is relatively straightforward. However, in subjective tasks such as role-playing, alignment techniques struggle to make progress, primarily because subjective reward modeling using the Bradley-Terry model faces significant challenges when dealing with ambiguous preferences. To improve reward modeling in subjective tasks, this paper proposes AAM (\textbf{\underline{A}}ct-\textbf{\underline{A}}daptive \textbf{\underline{M}}argin), which enhances reward modeling by dynamically calibrating preference margins using the model's internal parameter knowledge. We design two versions of AAM that efficiently generate contextually-appropriate preference gaps without additional human annotation. This approach fundamentally improves how reward models handle subjective rewards by better integrating generative understanding with preference scoring. To validate AAM's effectiveness in subjective reward modeling, we conduct evaluations on RewardBench, JudgeBench, and challenging role-playing tasks. Results show that AAM significantly improves subjective reward modeling performance, enhancing Bradley-Terry reward models by 2.95\% in general tasks and 4.85\% in subjective role-playing tasks. Furthermore, reward models trained with AAM can help downstream alignment tasks achieve better results. Our test results show that applying rewards generated by AAM-Augmented RM to preference learning techniques (e.g., GRPO) achieves state-of-the-art results on CharacterEval and Charm. Code and dataset are available at https://github.com/calubkk/AAM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑