arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01730cs.LGcs.ITmath.ITstat.ML
收藏

语言模型对齐的渐近理论

Asymptotics of Language Model Alignment

University of Sydney(悉尼大学);Massachusetts Institute of Technology(麻省理工学院);Google Research(谷歌研究院)

Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, Ahmad Beirami

AI总结:

本文从渐近与信息论角度分析语言模型对齐,给出 KL 约束强化学习最优解的闭式刻画,并证明 best-of-N 方法在期望奖励和 KL 散度意义上与其渐近等价。

摘要:

设 $p$ 表示一个生成式语言模型。设 $r$ 表示一个奖励模型,它返回一个标量,用于刻画从 $p$ 中采样得到的结果被偏好的程度。语言模型对齐的目标是将 $p$ 改变为一个新的分布 $ϕ$,从而获得更高的期望奖励,同时保持 $ϕ$ 与 $p$ 接近。一种流行的对齐方法是 KL 约束强化学习(RL),它选择一个分布 $ϕ_Δ$,在相对熵约束 $KL(ϕ_Δ|| p) \leq Δ$ 下最大化 $E_{ϕ_Δ} r(y)$。另一种简单的对齐方法是 best-of-$N$,即从 $p$ 中采样 $N$ 个样本,并选择其中奖励最高的一个。在本文中,我们给出了最优 KL 约束 RL 解的闭式刻画。我们证明,任何在 KL 散度与奖励之间实现可比权衡的对齐方法,都必须在相对熵意义上逼近最优 KL 约束 RL 解。为了进一步分析对齐方法的性质,我们引入两个简化假设:令语言模型是无记忆的,并令奖励模型是线性的。尽管这些假设可能无法反映复杂的现实场景,但它们使我们能够从信息论量的角度精确刻画 best-of-$N$ 对齐和 KL 约束 RL 方法的渐近行为。我们证明,最优 KL 约束 RL 解的奖励满足大偏差原理,并完整刻画了其速率函数。我们还证明,奖励的缩放累积量的增长率由一个适当的 Renyi 交叉熵来刻画。最后,我们证明 best-of-$N$ 与 KL 约束 RL 解渐近等价:通过证明它们的期望奖励渐近相等,并得出这两个分布必然在 KL 散度上彼此接近的结论。

英文摘要:

Let $p$ denote a generative language model. Let $r$ denote a reward model that returns a scalar that captures the degree at which a draw from $p$ is preferred. The goal of language model alignment is to alter $p$ to a new distribution $ϕ$ that results in a higher expected reward while keeping $ϕ$ close to $p.$ A popular alignment method is the KL-constrained reinforcement learning (RL), which chooses a distribution $ϕ_Δ$ that maximizes $E_{ϕ_Δ} r(y)$ subject to a relative entropy constraint $KL(ϕ_Δ|| p) \leq Δ.$ Another simple alignment method is best-of-$N$, where $N$ samples are drawn from $p$ and one with highest reward is selected. In this paper, we offer a closed-form characterization of the optimal KL-constrained RL solution. We demonstrate that any alignment method that achieves a comparable trade-off between KL divergence and reward must approximate the optimal KL-constrained RL solution in terms of relative entropy. To further analyze the properties of alignment methods, we introduce two simplifying assumptions: we let the language model be memoryless, and the reward model be linear. Although these assumptions may not reflect complex real-world scenarios, they enable a precise characterization of the asymptotic behavior of both the best-of-$N$ alignment, and the KL-constrained RL method, in terms of information-theoretic quantities. We prove that the reward of the optimal KL-constrained RL solution satisfies a large deviation principle, and we fully characterize its rate function. We also show that the rate of growth of the scaled cumulants of the reward is characterized by a proper Renyi cross entropy. Finally, we show that best-of-$N$ is asymptotically equivalent to KL-constrained RL solution by proving that their expected rewards are asymptotically equal, and concluding that the two distributions must be close in KL divergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑