arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.04834cs.CL
收藏

如同微调一样简单:通过双向负反馈损失实现 LLM 对齐

As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss

Nanyang Technological University(南洋理工大学);Shopee Pte. Ltd(Shopee私人有限公司);SEA Group(SEA集团)

Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Wang Chen, Anh Tuan Luu

AI总结:

本文提出双向负反馈损失 BF,以无需成对数据或额外超参数的类微调流程稳定 LLM 对齐,并在 QA 与推理基准上更好兼顾价值对齐和推理能力。

摘要:

直接偏好优化(Direct Preference Optimization,DPO)已成为采用近端策略优化(Proximal Policy Optimization,PPO)的人类反馈强化学习(Reinforcement Learning from Human Feedback,RLHF)的一种计算效率更高的替代方案,它消除了对奖励模型和在线采样的需求。尽管有这些优势,DPO 及其变体仍对超参数敏感,并且容易出现不稳定,尤其是在数学数据集上。我们认为,这些问题源于对数似然损失函数中固有的单向似然导数负反馈。为解决这一问题,我们提出了一种新颖的 LLM 对齐损失,可在优化过程中建立稳定的双向负反馈(Bidirectional Negative Feedback,BNF)。我们提出的 BNF 损失无需成对对比损失,也不需要任何额外可调超参数或成对偏好数据,从而将对齐流程简化得如同监督微调一样简单。我们在两个具有挑战性的 QA 基准和四个推理基准上进行了大量实验。实验结果表明,BNF 在 QA 基准上取得了与最佳方法相当的性能,而其在四个推理基准上的性能下降相比最佳方法显著更小,因此在价值对齐与推理能力之间取得了更好的平衡。此外,我们进一步验证了 BNF 在非成对数据集上的性能,并对不同偏好优化方法中的对数似然和 logit 偏移进行了深入分析。

英文摘要:

Direct Preference Optimization (DPO) has emerged as a more computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO), eliminating the need for reward models and online sampling. Despite these benefits, DPO and its variants remain sensitive to hyper-parameters and prone to instability, particularly on mathematical datasets. We argue that these issues arise from the unidirectional likelihood-derivative negative feedback inherent in the log-likelihood loss function. To address this, we propose a novel LLM alignment loss that establishes a stable Bidirectional Negative Feedback (BNF) during optimization. Our proposed BNF loss eliminates the need for pairwise contrastive losses and does not require any extra tunable hyper-parameters or pairwise preference data, streamlining the alignment pipeline to be as simple as supervised fine-tuning. We conduct extensive experiments across two challenging QA benchmarks and four reasoning benchmarks. The experimental results show that BNF achieves comparable performance to the best methods on QA benchmarks, while its performance decrease on the four reasoning benchmarks is significantly lower compared to the best methods, thus striking a better balance between value alignment and reasoning ability. In addition, we further validate the performance of BNF on non-pairwise datasets, and conduct in-depth analysis of log-likelihood and logit shifts across different preference optimization methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑