个性化奖励基准：评估与人类对齐的个性化
Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
AI总结：
本文提出个性化奖励基准，用于评估奖励模型对个性化偏好的建模能力，发现现有模型在个性化任务中表现不佳，且该基准在下游任务中具有更高的预测相关性。
摘要：
多元对齐已成为大型语言模型（LLM）发展中的关键前沿，奖励模型（RM）是捕捉多样化人类价值观的核心机制。尽管通用响应质量的基准广泛存在，但评估奖励模型如何反映个体用户偏好仍是一个开放挑战。为此，我们引入了个性化奖励基准，一种新的基准，用于严格评估奖励模型对个性化偏好的建模能力。我们基于严格遵守（或违反）用户特定标准构建了选择和拒绝的响应对，确保偏好区别独特地针对个体。特别是，人类评估确认了配对之间的主要区分因素是严格个人偏好，且两个响应保持高质量（例如正确性、相关性和帮助性）。广泛测试显示，现有最先进的奖励模型在个性化方面表现显著不佳，峰值准确率为仅75.94%。关键的是，因为有效的奖励模型基准应预测奖励模型在下游任务中的表现，我们进行了实验，证明我们的基准在最佳N（BoN）采样和近端策略优化（PPO）中与下游表现的相关性显著高于现有基线。这些发现确立了个性化奖励基准作为评估奖励模型在下游应用性能的稳健且准确的代理。
英文摘要：
Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
