arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.00747cs.LGcs.AI
收藏

软偏好优化:将语言模型对齐到专家分布

Soft Preference Optimization: Aligning Language Models to Expert Distributions

University of Alberta(阿尔伯塔大学);Openmind Research Institute(Openmind研究所);Leiden University(莱顿大学);Spotify;Google DeepMind(谷歌DeepMind)

Arsalan Sharifnassab, Saber Salehkaleybar, Sina Ghiassian, Surya Kanoria, Dale Schuurmans

AI总结:

本文提出软偏好优化(SPO),无需奖励模型即可将大语言模型对齐到人类偏好,通过结合偏好损失与全输出分布正则化实现高效对齐,并证明其在 Bradley-Terry 假设下收敛到可调软度的 softmax 奖励分布。

摘要:

我们提出软偏好优化(Soft Preference Optimization,SPO),这是一种无需奖励模型即可将生成模型(如大语言模型,LLMs)与人类偏好对齐的方法。SPO 通过一个自然损失函数直接在偏好数据集上优化模型输出,该损失函数将偏好损失与一个正则化项结合,作用于模型在整个输出分布上的表现,而非仅限于偏好数据集。尽管 SPO 不要求存在底层奖励模型的假设,我们证明在 Bradley-Terry(BT)模型假设下,它收敛到缩放奖励的 softmax,分布的“软度”可通过 softmax 指数这一算法参数调节。我们展示了 SPO 的方法论、理论基础,以及其在简洁性、计算效率和对齐精度方面的比较优势。

英文摘要:

We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑