arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.14836cs.CLcs.AIcs.LG
收藏

直接偏好优化的正交微调

Orthogonal Finetuning for Direct Preference Optimization

Chenxu Yang, Ruipeng Jia, Naibin Gu, Zheng Lin, Siyuan Chen, Chao Pang, Weichong Yin, Yu Sun, Hua Wu, Weiping Wang

AI总结:

针对DPO微调模型过拟合于不受偏好样本的问题,本文提出RoPO正交微调方法,通过保持超球面能量不变抑制过拟合,仅用0.0086%可训练参数提升对齐性能与生成多样性,在MT-Bench和AlpacaEval 2上均优于DPO。

摘要:

DPO是一种有效的偏好优化算法。然而,经DPO微调的模型往往会过拟合于不受偏好的样本,表现为生成内容过长且缺乏多样性。尽管近期的正则化方法尝试通过修改目标函数来缓解这一问题,但它们是以对齐性能下降为代价实现的。在本文中,我们创新性地从权重更新的角度引入正则化,以抑制对齐过拟合。通过初步实验,我们发现过拟合与超球面能量波动之间存在正相关关系。因此,我们通过一种权重旋转偏好优化(RoPO)方法为DPO引入正交微调,该方法仅对权重参数进行旋转和幅度拉伸更新,以保持超球面能量不变,从而保留神经元之间角度所编码的知识。大量实验表明,我们的模型仅使用0.0086%的可训练参数,就能完美对齐人类偏好并保留原始表达能力,这表明该方法是一种有效的抗过拟合正则化。具体而言,RoPO在MT-Bench上比DPO高出10分,在AlpacaEval 2上高出2.8分,同时生成多样性平均提升6分。

英文摘要:

DPO is an effective preference optimization algorithm. However, the DPO-tuned models tend to overfit on the dispreferred samples, manifested as overly long generations lacking diversity. While recent regularization approaches have endeavored to alleviate this issue by modifying the objective function, they achieved that at the cost of alignment performance degradation. In this paper, we innovatively incorporate regularization from the perspective of weight updating to curb alignment overfitting. Through the pilot experiment, we discovered that there exists a positive correlation between overfitting and the hyperspherical energy fluctuation. Hence, we introduce orthogonal finetuning for DPO via a weight-Rotated Preference Optimization (RoPO) method, which merely conducts rotational and magnitude-stretching updates on the weight parameters to maintain the hyperspherical energy invariant, thereby preserving the knowledge encoded in the angle between neurons. Extensive experiments demonstrate that our model aligns perfectly with human preferences while retaining the original expressive capacity using only 0.0086% of the trainable parameters, suggesting an effective regularization against overfitting. Specifically, RoPO outperforms DPO by up to 10 points on MT-Bench and by up to 2.8 points on AlpacaEval 2, while enhancing the generation diversity by an average of 6 points.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑