arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.19733cs.CLcs.AI
收藏

迭代推理偏好优化

Iterative Reasoning Preference Optimization

FAIR at Meta(Meta FAIR研究院);New York University(纽约大学)

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, Jason Weston

AI总结:

提出迭代推理偏好优化方法,通过修改DPO损失并加入负对数似然项优化思维链推理步骤偏好,在GSM8K等基准上显著提升Llama-2-70B-Chat准确率。

摘要:

迭代偏好优化方法近来已被证明在通用指令微调任务上表现良好,但在推理任务上通常改进甚微(Yuan等,2024;Chen等,2024)。在本工作中,我们开发了一种迭代方法,通过优化导致正确答案的获胜推理步骤与失败推理步骤之间的偏好,来优化竞争性生成的思维链(Chain-of-Thought, CoT)候选之间的偏好。我们使用修改后的DPO损失(Rafailov等,2023)进行训练,并额外加入负对数似然项,我们发现这一项至关重要。我们展示了在重复迭代此方案时推理能力得到提升。虽然仅依赖训练集中的示例,我们的方法在GSM8K、MATH和ARC-Challenge上对Llama-2-70B-Chat的准确率有所提升,超越了其他不依赖额外来源数据集的基于Llama-2的模型。例如,我们在GSM8K上看到了从55.6%到81.6%的大幅提升,并且在32个样本的多数投票中达到了88.7%的准确率。

英文摘要:

Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps that lead to the correct answer. We train using a modified DPO loss (Rafailov et al., 2023) with an additional negative log-likelihood term, which we find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy on GSM8K, MATH, and ARC-Challenge for Llama-2-70B-Chat, outperforming other Llama-2-based models not relying on additionally sourced datasets. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑