ASFT:通过绝对似然实现的对齐监督微调
ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood
AI总结:
针对DPO依赖SFT效果、对齐人类偏好存在局限的问题,提出无需参考模型、基于绝对似然优化的ASFT对齐方法,经理论分析与实验验证其性能优于DPO等现有方法。
摘要:
直接偏好优化(Direct Preference Optimization, DPO)是一种通过直接针对结果的偏好或排名进行优化来提升模型性能的方法,无需使用传统损失函数。该方法已被证明在使大语言模型(Large Language Models, LLMs)对齐人类偏好方面效果显著。尽管DPO已在各类任务中得到广泛应用,但它仍饱受批评:其性能高度依赖监督微调(Supervised Fine-Tuning, SFT)的效果,且在引导模型学习人类偏好的回复方面存在局限,最终导致表现不尽如人意。\n为解决这些局限,我们提出了对齐监督微调(Aligned Supervised Fine-Tuning, ASFT)——一种有效的对齐方法。它通过优化每个回复的绝对似然(而非使用Bradley-Terry模型),让LLM更好地对齐成对偏好数据集,且无需参考模型。通过理论梯度分析,我们证明ASFT缓解了DPO损失函数的一个问题:DPO降低生成人类非偏好数据概率的速度,快于其提升生成偏好数据概率的速度。\n此外,我们使用在UltraFeedback和HH-RLHF上微调的最新指令调优模型Llama3,将ASFT与DPO及其最新变体(如单步方法ORPO)进行了对比。我们在MT-Bench等指令遵循基准,以及BLEU-4、ROUGE-L等传统文本生成指标上评估了性能。大量实验表明,ASFT是一种有效的对齐方法,表现始终优于现有方法。
英文摘要:
Direct Preference Optimization (DPO) is a method for enhancing model performance by directly optimizing for the preferences or rankings of outcomes, instead of traditional loss functions. This approach has proven effective in aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the effectiveness of Supervised Fine-Tuning (SFT) and its limitations in enabling models to learn human-preferred responses, leading to less satisfactory performance. To address these limitations, we propose Aligned Supervised Fine-Tuning (ASFT), an effective approach that better aligns LLMs with pair-wise datasets by optimizing absolute likelihood for each response, rather than using the Bradley-Terry model, and eliminates the need for a reference model. Through theoretical gradient analysis, we demonstrate that ASFT mitigates the issue where the DPO loss function decreases the probability of generating human-dispreferred data at a faster rate than it increases the probability of producing preferred data. Additionally, we compare ASFT to DPO and its latest variants, such as the single-step approach ORPO, using the latest instruction-tuned model Llama3, which has been fine-tuned on UltraFeedback and HH-RLHF. We evaluated performance on instruction-following benchmarks like MT-Bench and traditional text generation metrics such as BLEU-4 and ROUGE-L. Extensive experiments demonstrate that ASFT is an effective alignment approach, consistently outperforming existing methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。