变换与组合奖励以对齐大型语言模型
Transforming and Combining Rewards for Aligning Large Language Models
AI总结：
本研究针对奖励模型对齐语言模型中的变换与组合问题，提出LSC变换（log-sigmoid中心化变换），通过强调改进差输出并实现原则性聚合，在RLHF实验中显著提升有用性与无害性。
摘要：
将语言模型与人类偏好对齐的一种常见方法是，首先从偏好数据中学习一个奖励模型，然后使用该奖励模型来更新语言模型。我们研究了这种方法中出现的两个密切相关的问题。第一，奖励模型的任何单调变换都保持偏好排序；是否存在一个比其他选择“更好”的选择？第二，我们常常希望将语言模型与多个属性对齐：我们应该如何组合多个奖励模型？利用对齐过程的概率解释，我们为（常见情况下的）从Bradley-Terry偏好模型学习的奖励确定了一个自然的变换选择。推导出的变换很直接：我们对中心化奖励应用log-sigmoid函数，我们将此方法称为“LSC变换”（log-sigmoid中心化变换）。该变换具有两个重要性质。第一，它强调改进表现不佳的输出，而不是已经得分较高的输出。这既缓解了欠拟合（某些提示未被改进）也缓解了奖励黑客（模型学会利用奖励模型的错误设定）。第二，它通过将求和与逻辑合取联系起来，实现了奖励的原则性聚合：变换后奖励的总和对应于输出在所有测量属性中“良好”的概率，我们对此进行了精确阐述。使用RLHF将语言模型对齐为既有用又无害的实验显示，与基线（非变换）方法相比有显著改进。
英文摘要：
A common approach for aligning language models to human preferences is to first learn a reward model from preference data, and then use this reward model to update the language model. We study two closely related problems that arise in this approach. First, any monotone transformation of the reward model preserves preference ranking; is there a choice that is ``better'' than others? Second, we often wish to align language models to multiple properties: how should we combine multiple reward models? Using a probabilistic interpretation of the alignment procedure, we identify a natural choice for transformation for (the common case of) rewards learned from Bradley-Terry preference models. The derived transformation is straightforward: we apply a log-sigmoid function to the centered rewards, a method we term ``LSC-transformation'' (log-sigmoid-centered transformation). This transformation has two important properties. First, it emphasizes improving poorly-performing outputs, rather than outputs that already score well. This mitigates both underfitting (where some prompts are not improved) and reward hacking (where the model learns to exploit misspecification of the reward model). Second, it enables principled aggregation of rewards by linking summation to logical conjunction: the sum of transformed rewards corresponds to the probability that the output is ``good'' in all measured properties, in a sense we make precise. Experiments aligning language models to be both helpful and harmless using RLHF show substantial improvements over the baseline (non-transformed) approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
