PersRM-R1：通过强化学习增强个性化奖励建模
PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning
AI总结：
针对现有奖励模型（RMs）难以捕捉有限数据下用户特定偏好的问题，提出首个基于推理的PersRM-R1框架，结合合成数据生成与两阶段训练（监督+强化微调），实现从少量示例识别个人因素，性能优于同规模模型且比肩大模型。
摘要：
奖励模型（RMs）是现有后训练方法的核心，旨在通过在微调期间提供反馈信号使LLM输出与人类价值观对齐。然而，现有RMs难以捕捉细致的用户特定偏好，尤其在有限数据和不同领域下。因此，我们引入PersRM-R1，首个基于推理的奖励建模框架，专门设计用于从仅一个或几个个人示例中识别和表示个人因素。为解决有限数据可用性和鲁棒泛化需求等挑战，我们的方法结合合成数据生成与两阶段训练流程——监督微调后接强化微调。实验结果表明，PersRM-R1在准确性和泛化性上优于类似规模现有模型，且与更大模型性能相当，为更有效的个性化LLMs铺路。
英文摘要：
Reward models (RMs), which are central to existing post-training methods, aim to align LLM outputs with human values by providing feedback signals during fine-tuning. However, existing RMs struggle to capture nuanced, user-specific preferences, especially under limited data and across diverse domains. Thus, we introduce PersRM-R1, the first reasoning-based reward modeling framework specifically designed to identify and represent personal factors from only one or a few personal exemplars. To address challenges including limited data availability and the requirement for robust generalization, our approach combines synthetic data generation with a two-stage training pipeline consisting of supervised fine-tuning followed by reinforcement fine-tuning. Experimental results demonstrate that PersRM-R1 outperforms existing models of similar size and matches the performance of much larger models in both accuracy and generalizability, paving the way for more effective personalized LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
