arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.14723cs.CL
收藏

对齐的洞察:评估DPO及其变体在多个任务上的表现

Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks

Arizona State University(亚利桑那州立大学)

Amir Saeidi, Shivanshu Verma, Md Nayem Uddin, Chitta Baral

AI总结:

本文评估DPO及其变体在三种配置下对齐LLM的效果,发现小数据子集即可接近最优性能,对齐方法虽对复杂推理提升有限但能增强数学求解,且指令微调模型可提高真实性。

摘要:

本研究评估了直接偏好优化(DPO)及其变体在使大型语言模型(LLMs)与人类偏好对齐方面的表现,测试了三种配置:(1)使用监督微调(SFT),(2)不使用SFT,以及(3)不使用SFT但使用指令微调模型。我们进一步研究了训练集大小如何影响模型性能。我们的评估涵盖13个基准,包括对话、推理、数学问题求解、问答、真实性、MT-Bench、Big Bench和Open LLM Leaderboard。我们发现:(1)即使使用较小的训练数据子集,对齐方法通常也能达到接近最优的性能;(2)尽管它们在复杂推理任务上的改进有限,但它们增强了数学问题求解能力;(3)使用指令微调模型可以提高真实性。这些见解突出了对齐方法表现出色的条件及其局限性。

英文摘要:

This study evaluates Direct Preference Optimization (DPO) and its variants for aligning Large Language Models (LLMs) with human preferences, testing three configurations: (1) with Supervised Fine Tuning (SFT), (2) without SFT, and (3) without SFT but using an instruction tuned model. We further investigate how training set size influences model performance. Our evaluation spans 13 benchmarks covering dialogue, reasoning, mathematical problem-solving, question answering, truthfulness, MT-Bench, Big Bench, and the Open LLM Leaderboard. We find that: (1) alignment methods often achieve near optimal performance even with smaller subsets of training data; (2) although they offer limited improvements on complex reasoning tasks, they enhance mathematical problem-solving; and (3) using an instruction tuned model improves truthfulness. These insights highlight the conditions under which alignment methods excel, as well as their limitations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑