重新思考基于偏好的奖励建模中的 Bradley-Terry 模型：基础、理论与替代方案
Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives
AI总结：
本文重新审视了LLM奖励建模中BT模型的理论基础，提出顺序一致性概念及兼容二分类器的上界算法作为替代方案，并在超12000个实验设置中验证了不同方法的效果。
摘要：
Bradley-Terry (BT) 模型是大语言模型 (LLM) 对齐奖励建模中常见且成功的实践。然而，目前尚不清楚为什么这个最初为多人随机博弈匹配开发的模型能够被用于将成对响应比较转换为奖励值并进行预测。特别是考虑到只有有限数量的提示-响应对与其他对进行了稀疏比较这一事实。在本文中，我们首先重新审视了在奖励建模中使用 BT 模型的基础，并基于使用嵌入的深度神经网络建立了 BT 奖励模型的收敛速率，为其使用提供了理论基础。尽管在理论上是合理的，我们论证了从下游优化的角度来看，BT 模型并非必要的选择。这是因为奖励模型只需要通过对真实奖励的单调变换来保持正确的排序预测。我们强调了奖励建模中顺序一致性的关键概念，并证明 BT 模型具备这一性质。因此，我们提出了一种简单直接的基于上界的算法，兼容现成的二分类器，作为替代的顺序一致奖励建模目标。为了提供实践见解，我们实证评估了这些不同奖励建模方法的性能，涵盖超过 12,000 个实验设置，使用了 6 个基础 LLM、2 个数据集，以及在偏好标注的数量、质量和配对选择上有所不同的多样化标注设计。
英文摘要：
The Bradley-Terry (BT) model is a common and successful practice in reward modeling for Large Language Model (LLM) alignment. However, it remains unclear why this model -- originally developed for multi-player stochastic game matching -- can be adopted to convert pairwise response comparisons to reward values and make predictions. Especially given the fact that only a limited number of prompt-response pairs are sparsely compared with others. In this paper, we first revisit the foundations of using BT models in reward modeling, and establish the convergence rate of BT reward models based on deep neural networks using embeddings, providing a theoretical foundation for their use. Despite theoretically sound, we argue that the BT model is not a necessary choice from the perspective of downstream optimization. This is because a reward model only needs to preserve the correct ranking predictions through a monotonic transformation of the true reward. We highlight the critical concept of order consistency in reward modeling and demonstrate that the BT model possesses this property. Consequently, we propose a simple and straightforward upper-bound algorithm, compatible with off-the-shelf binary classifiers, as an alternative order-consistent reward modeling objective. To offer practical insights, we empirically evaluate the performance of these different reward modeling approaches across more than 12,000 experimental setups, using $6$ base LLMs, $2$ datasets, and diverse annotation designs that vary in quantity, quality, and pairing choices in preference annotations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
