每个人都应获得奖励：学习定制化的人类偏好
Everyone Deserves A Reward: Learning Customized Human Preferences
AI总结：
针对通用奖励模型无法满足个性化需求的问题，提出三阶段定制化奖励模型学习方案，并构建领域特定偏好数据集，发现通用偏好增强与定制化偏好模仿学习能有效保留通用偏好能力。
摘要：
奖励模型（RMs）对于使大型语言模型（LLMs）与人类偏好对齐以提升交互质量至关重要。然而，现实世界是多元的，这导致在宗教、政治、文化等方面存在多样化的人类偏好。此外，每个个体在各类主题上也可能拥有独特的偏好。当前的人类反馈对齐方法忽略了人类偏好的多样性，仅考虑通用的奖励模型，这在定制化或个性化应用场景中表现不尽如人意。为了探索定制化偏好学习，我们收集了一个领域特定偏好（DSP）数据集，其中包含来自四个实际领域的每个给定查询的首选响应。此外，从数据效率的角度出发，我们提出了一种三阶段定制化奖励模型学习方案，随后在通用偏好数据集和我们的DSP数据集上实证验证了其有效性。进一步地，我们在三个学习阶段测试了多种训练与数据策略。我们发现了在训练定制化奖励模型时能更好地保留通用偏好能力的几种方法，尤其是通用偏好增强和定制化偏好模仿学习。DSP数据集和代码可在 https://github.com/Linear95/DSP 获取。
英文摘要：
Reward models (RMs) are essential for aligning large language models (LLMs) with human preferences to improve interaction quality. However, the real world is pluralistic, which leads to diversified human preferences with respect to different religions, politics, cultures, etc. Moreover, each individual can have their unique preferences on various topics. Neglecting the diversity of human preferences, current human feedback aligning methods only consider a general reward model, which is below satisfaction for customized or personalized application scenarios. To explore customized preference learning, we collect a domain-specific preference (DSP) dataset, which includes preferred responses for each given query from four practical domains. Besides, from the perspective of data efficiency, we propose a three-stage customized RM learning scheme, then empirically verify its effectiveness on both general preference datasets and our DSP set. Furthermore, we test multiple training and data strategies on the three learning stages. We find several ways to better preserve the general preferring ability while training the customized RMs, especially general preference enrichment, and customized preference imitation learning. The DSP dataset and code are available at https://github.com/Linear95/DSP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
