通过最大边际似然估计实现离线偏好优化
Offline Preference Optimization via Maximum Marginal Likelihood Estimation
AI总结：
本文提出基于最大边际似然估计的MMPO方法，通过隐式偏好优化提升模型稳定性与对齐性能。
摘要：
将大型语言模型（LLMs）与人类偏好对齐是至关重要的，但标准方法如基于人类反馈的强化学习（RLHF）往往复杂且不稳定。在本工作中，我们提出了一种新的、更简单的方法，通过最大边际似然（MML）估计的视角来重新审视对齐问题。我们新的基于MML的偏好优化（MMPO）通过将首选文本输出的边际对数似然最大化，利用偏好对作为近似样本，并省去了显式奖励模型和熵最大化的需求。我们理论证明MMPO隐式地执行偏好优化，产生一个加权梯度，自然地使选择响应相对于拒绝响应具有更高的权重。在从135M到8B参数的模型上，我们实证显示MMPO：1）相比其他基线在超参数β方面更加稳定，2）在实现竞争或更优的偏好对齐的同时，更好地保持了基础模型的语言能力。通过一系列消融实验，我们证明这种改进的性能确实归因于MMPO在梯度更新中的隐式偏好优化。
英文摘要：
Aligning Large Language Models (LLMs) with human preferences is crucial, but standard methods like Reinforcement Learning from Human Feedback (RLHF) are often complex and unstable. In this work, we propose a new, simpler approach that recasts alignment through the lens of Maximum Marginal Likelihood (MML) estimation. Our new MML based Preference Optimization (MMPO) maximizes the marginal log-likelihood of a preferred text output, using the preference pair as samples for approximation, and forgoes the need for both an explicit reward model and entropy maximization. We theoretically demonstrate that MMPO implicitly performs preference optimization, producing a weighted gradient that naturally up-weights chosen responses over rejected ones. Across models ranging from 135M to 8B parameters, we empirically show that MMPO: 1) is more stable with respect to the hyperparameter $β$ compared to alternative baselines, and 2) achieves competitive or superior preference alignment while better preserving the base model's general language capabilities. Through a series of ablation experiments, we show that this improved performance is indeed attributable to MMPO's implicit preference optimization within the gradient updates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
