arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.07230cs.CLcs.AIcs.LG
收藏

Curry-DPO:利用课程学习与排序偏好增强对齐

Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences

ServiceNow(ServiceNow公司)

Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, Sathwik Tejaswi Madhusudhan

AI总结:

Curry-DPO通过课程学习将多偏好对按难度排序用于DPO训练,在多个基准上显著提升对齐性能,最高胜率提升达7.5%。

摘要:

直接偏好优化(DPO)是一种有效的技术,利用成对偏好数据(通常每个用户提示对应一个被选中的响应和一个被拒绝的响应)来使大型语言模型(LLM)与人类偏好对齐。在实践中,对于给定的提示,可能存在多个质量相对各异的响应。在可获得多个响应的质量评级的情况下,我们提出利用这些响应为给定提示创建多个偏好对。我们的工作重点在于通过课程学习方法论,系统地使用构建的多个偏好对进行DPO训练。具体而言,我们根据各种标准,将这些多个偏好对从易到难排序(模拟课程训练)。我们展示了我们提出的方法与标准单对DPO设置的详细比较。我们称之为Curry-DPO的方法在MT-bench、Vicuna、WizardLM和UltraFeedback测试集上持续显示出性能提升,突显了其有效性。更具体地说,Curry-DPO在Zephy-7B模型上于MT-bench取得了7.43的分数,超越了大多数具有相似参数规模的现有LLM。在我们的实验中,Curry-DPO还在Vicuna、WizardLM和UltraFeedback测试数据集上取得了最高的调整后胜率(分别为90.7%、87.1%和87.9%),与标准DPO技术相比,显著提升了高达7.5%。我们在https://huggingface.co/datasets/ServiceNow-AI/Curriculum_DPO_preferences发布了对齐中使用的偏好对。

英文摘要:

Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data (usually one chosen and rejected response pair per user prompt) to align LLMs to human preferences. In practice, multiple responses can exist for a given prompt with varying quality relative to each other. With availability of such quality ratings for multiple responses, we propose utilizing these responses to create multiple preference pairs for a given prompt. Our work focuses on systematically using the constructed multiple preference pair in DPO training via curriculum learning methodology. In particular, we order these multiple pairs of preference data from easy to hard (emulating curriculum training) according to various criteria. We show detailed comparisons of our proposed approach to the standard single-pair DPO setting. Our method, which we call Curry-DPO consistently shows increased performance gains on MTbench, Vicuna, WizardLM, and the UltraFeedback test set, highlighting its effectiveness. More specifically, Curry-DPO achieves a score of 7.43 on MT-bench with Zephy-7B model outperforming majority of existing LLMs with similar parameter size. Curry-DPO also achieves the highest adjusted win rates on Vicuna, WizardLM, and UltraFeedback test datasets (90.7%, 87.1%, and 87.9% respectively) in our experiments, with notable gains of upto 7.5% when compared to standard DPO technique. We release the preference pairs used in alignment at: https://huggingface.co/datasets/ServiceNow-AI/Curriculum_DPO_preferences

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑