arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.05749cs.LGcs.AIstat.ML
收藏

广义偏好优化:离线对齐的统一方法

Generalized Preference Optimization: A Unified Approach to Offline Alignment

Google DeepMind(谷歌DeepMind)

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, Bilal Piot

AI总结:

本文提出广义偏好优化(GPO)框架,统一了DPO、IPO和SLiC等离线偏好优化算法,并揭示了离线正则化与KL散度正则化的联系,为对齐实践提供了新工具和经验见解。

摘要:

离线偏好优化允许直接从离线数据微调大型模型,并在最近的对齐实践中被证明是有效的。我们提出了广义偏好优化(GPO),这是一族由一般凸函数类参数化的离线损失函数。GPO 为偏好优化提供了统一视角,将现有算法如 DPO、IPO 和 SLiC 作为特例包含在内,同时自然引入新的变体。GPO 框架还揭示了离线算法如何通过定义损失的凸函数设计来强制执行正则化。我们的分析和实验揭示了离线正则化与经典 RLHF 公式所意图的 KL 散度正则化之间的联系和细微差异。在与 Gao 等人 2023 类似的可控设置中,我们还表明不同的 GPO 变体在正则化和性能之间实现类似的权衡,尽管超参数的最优值可能如理论所预测的那样不同。总之,我们的结果为对齐实践者提供了新的算法工具包和经验见解。

英文摘要:

Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al 2023, we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑