arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12116cs.CL
收藏

P-GenRM:具有测试时用户基于缩放的个性化生成奖励模型

P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling

Alibaba Group(阿里巴巴集团)

Pinyi Zhang, Ting-En Lin, Yuchuan Wu, Jingyang Chen, Zongqi Wang, Hua Yang, Ze Xu, Fei Huang, Kai Zhang, Yongbin Li

AI总结:

P-GenRM通过测试时用户基于缩放机制,实现个性化生成奖励模型,提升用户偏好适应性和泛化能力。

摘要:

个性化对齐大型语言模型旨在将响应适应于个体用户偏好,通常通过强化学习实现。关键挑战是获得准确的、用户特定的奖励信号在开放性场景中。现有的个性化奖励模型面临两个持续存在的局限性:(1)将多样、场景特定的偏好简化为一小套固定的评估原则;(2)难以泛化到有有限反馈的新用户。为此,我们提出了P-GenRM,第一个具有测试时用户基于缩放的个性化生成奖励模型。P-GenRM将偏好信号转换为结构化的评估链,从而在各种场景中推导出适应性的人格和评分标准。它进一步将用户聚类为用户原型,并引入了双粒度缩放机制:在个体层面,它自适应地缩放和聚合每个用户的评分方案;在原型层面,它纳入了相似用户的偏好。这种设计通过原型转移减少了推断偏好的噪声,并通过原型基于的泛化增强了对未见用户的一般化能力。实证结果表明,P-GenRM在广泛使用的个性化奖励模型基准上实现了最先进的结果,平均改进了2.31%,并在分布外数据集上展示了强大的泛化能力。值得注意的是,测试时用户基于缩放提供了额外的3%提升,展示了更强的个性化对齐与测试时可扩展性。

英文摘要:

Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑