PAFO: 个性化奖励建模的帕累托公平优化
PAFO: Pareto Fairness Optimization for Personalized Reward Modeling
AI总结：
针对个性化奖励模型因训练数据偏好不平衡导致对少数用户群体存在偏见的问题，提出PAFO框架，通过帕累托公平优化提升弱势群体性能而不损害其他群体，实验表明能同时提高少数和多数群体准确率并降低不公平性。
摘要：
大型语言模型（LLMs）越来越依赖奖励模型来使其输出与多样化的用户偏好对齐。虽然个性化奖励模型旨在捕捉这种异质性，但它们通常在用户偏好数据不平衡的情况下训练，因此可能偏向于在训练群体中偏好更常见的用户。在本文中，我们将这种失败模式识别为个性化奖励偏差，即奖励建模质量随偏好支持率系统性地变化。我们将其缓解表述为一个关于群体效用的帕累托公平问题，旨在改善服务不足的用户而不降低其他用户群体的性能。为此，我们提出了PAFO，一种用于个性化奖励建模的帕累托公平优化框架。PAFO首先为多数和少数偏好群体训练群体专用的奖励模型，然后构建条件边际级监督，将其异质性偏好边界蒸馏到一个统一的模型中。所得模型仅在训练时使用群体信息，推理时无需显式群体标签。在Personal-LLM和DSP上的实验表明，PAFO在多个指标上提高了少数群体和多数群体的准确率，同时减少了用户级不公平性，证明了其在更公平的LLM个性化中的有效性。
英文摘要：
Large language models (LLMs) increasingly rely on reward models to align their outputs with diverse user preferences. While personalized reward models aim to capture such heterogeneity, they are often trained on imbalanced user preference data and may therefore favor users whose preferences are more common in the training population. In this paper, we identify this failure mode as personalized reward bias, where reward modeling quality varies systematically with preference support rate. We formulate its mitigation as a Pareto fairness problem over group utilities, aiming to improve under-served users without degrading other user groups. To this end, we propose PAFO, a Pareto fairness optimization framework for personalized reward modeling. PAFO first trains group-specialized reward models for majority and minority preference groups, then constructs conditional margin-level supervision to distill their heterogeneous preference boundaries into a single unified model. The resulting model uses group information only during training and requires no explicit group labels at inference time. Experiments on Personal-LLM and DSP show that PAFO improves both minority-group and majority-group accuracy while reducing user-level unfairness across multiple metrics, demonstrating its effectiveness for fairer LLM personalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
