重新审视奖励优化的缩放定律
Revisiting scaling laws for reward optimization
- MIT（麻省理工学院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究提出奖励优化的缩放定律，性能随训练数据量与散度预算的平方根增长，并通过信息论模型和实验验证，揭示其与简单选择任务的类比。
AI中文摘要：
在AI对齐中，针对奖励模型优化的缩放定律已确定了性能如何依赖于优化努力——以相对于参考策略的KL散度预算来衡量。超过一定预算后，可能出现过度优化（或奖励黑客攻击）：因为我们针对代理奖励模型（不同于真实奖励）进行优化，性能可能趋于平稳或下降。自然，代理奖励的准确性取决于用于训练它的偏好数据量（通常以成对比较的形式）。然而，现有研究并未清晰地确定性能如何随训练数据量和散度预算共同缩放。我们的主要贡献是在此背景下提供一个经验上准确且理论上扎实的缩放定律。性能大致按$\Theta(\sqrt{\min\{\log(M),K\}})$缩放，其中$M$是训练数据中的比较次数，$K$是策略的散度预算。我们开发了一个信息论模型来确立这一上界，并通过构造性过程证明其紧密可达。基于此，我们使用真实世界的标注设置进行了广泛的经验评估，其中大型70B黄金奖励模型生成反馈数据，而代理奖励模型由能力较弱的模型（0.6B至4B）训练。我们的缩放定律提供了极好的拟合（R2从97%到99%），优于替代规格，并在模型大小、噪声和优化过程（best-of-$N$或策略倾斜）中保持稳健。我们的证据表明，奖励优化类似于一个令人惊讶的简单选择任务：使用嘈杂的偏好反馈从一系列IID高斯随机变量中进行选择。
英文摘要：
Scaling laws for optimization against reward models in AI alignment have pinned down how performance depends on optimization effort---measured by a KL-divergence budget relative to a reference policy. Beyond a certain budget, over-optimization (or reward hacking) can arise: because we optimize against a proxy reward model (distinct from true rewards), performance can plateau or degrade. Naturally, the proxy reward's accuracy depends on how much preference data (often in the form of pairwise comparisons) was used to train it. However, existing research does not cleanly identify how performance jointly scales with the amount of training data and the divergence budget. Our main contribution is to provide an empirically accurate and theoretically grounded scaling law in such context. Performance roughly scales as $Θ(\sqrt{\min\{\log(M),K\}})$, where $M$ is the number of comparisons in training data and $K$ is the policy's divergence budget. We develop an information-theoretic model to establish this upper bound and prove it is tightly achievable through a constructive procedure. Informed by this, we conduct extensive empirical evaluations using a real-world annotation setup, whereby a large 70B gold reward model generates feedback data and proxy reward models are trained from less capable models (0.6B to 4B). Our scaling law provides an excellent fit (R2 from 97\% to 99\%), outperforms alternative specifications, and remains robust across model sizes, noise, and optimization procedures (best-of-$N$ or policy tilting). Our evidence suggests that reward optimization is analogous to a surprisingly simple selection task: choosing from a sequence of IID Gaussian random variables using noisy preference feedback.
