TPO：利用多分支与多步偏好树对齐大型语言模型
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
AI总结：
本文提出树偏好优化（TPO），直接利用整个偏好树进行偏好列表排序学习，并引入自适应步骤奖励，在数学推理任务上优于DPO，提升LLM长链推理能力。
摘要：
在复杂推理任务（如数学推理）领域，近期的进展提出了使用直接偏好优化（DPO）来抑制不受欢迎响应的输出，从而增强大型语言模型（LLMs）的长链推理能力。为此，这些研究利用LLMs通过思维树（ToT）生成偏好树，并采样DPO算法所需的成对偏好响应。然而，基于二元偏好优化的DPO算法无法学习偏好树提供的具有不同偏好/不偏好程度的多个响应，导致偏好学习不完整。在这项工作中，我们引入了树偏好优化（TPO），它不从偏好树中采样成对偏好响应，而是在微调期间直接学习整个偏好树。具体来说，TPO将语言模型对齐形式化为偏好列表排序问题，其中策略可以更有效地从给定提示的响应排序偏好列表中学习。此外，为了进一步帮助LLMs识别长链推理中的判别性步骤并增加偏好列表中的相对奖励边际，TPO利用自适应步骤奖励来调整轨迹中每一步的奖励值，以进行细粒度的偏好优化。我们在数学推理任务上进行了大量实验来评估TPO。实验结果表明，在四个数据集上，TPO在五个公开的大型语言模型上始终优于DPO。我们的代码可在https://github.com/MrBlankness/TPO.git公开获取。
英文摘要：
In the domain of complex reasoning tasks, such as mathematical reasoning, recent advancements have proposed the use of Direct Preference Optimization (DPO) to suppress output of dispreferred responses, thereby enhancing the long-chain reasoning capabilities of large language models (LLMs). To this end, these studies employed LLMs to generate preference trees via Tree-of-thoughts (ToT) and sample the paired preference responses required by the DPO algorithm. However, the DPO algorithm based on binary preference optimization is unable to learn multiple responses with varying degrees of preference/dispreference that provided by the preference trees, resulting in incomplete preference learning. In this work, we introduce Tree Preference Optimization (TPO), that does not sample paired preference responses from the preference tree; instead, it directly learns from the entire preference tree during the fine-tuning. Specifically, TPO formulates the language model alignment as a Preference List Ranking problem, where the policy can potentially learn more effectively from a ranked preference list of responses given the prompt. In addition, to further assist LLMs in identifying discriminative steps within long-chain reasoning and increase the relative reward margin in the preference list, TPO utilizes Adaptive Step Reward to adjust the reward values of each step in trajectory for performing fine-grained preference optimization. We carry out extensive experiments on mathematical reasoning tasks to evaluate TPO. The experimental results indicate that TPO consistently outperforms DPO across five public large language models on four datasets. Our code is publicly available at https://github.com/MrBlankness/TPO.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
