推进工具增强型大语言模型：整合推理树中错误带来的洞见
Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
AI总结：
针对现有工具增强LLM仅用推理树成功路径训练的局限，提出基于偏好学习的推理轨迹优化框架，构建分步偏好数据并结合DPO训练得到TP-LLaMA，性能、泛化性和推理效率均显著提升。
摘要：
工具增强型大语言模型（LLM）利用通常以API形式存在的工具来提升其在复杂任务上的推理能力，这使它们能够充当与真实世界交互的智能体。Qin等人[2023]近期提出的ToolLLaMA模型采用基于深度优先搜索的决策树（DFSDT）机制，结合16000多个真实世界API实现多步推理，与传统链式推理机制相比，有效提升了工具增强型LLM的性能。然而，他们的方法仅使用决策树（也称为推理树）中的成功路径进行监督微调（SFT），错失了从失败路径中获取潜在学习机会的可能。受此启发，我们提出了一种基于偏好学习的推理轨迹优化框架来解决这一局限。我们首先提出了一种从树状专家轨迹中构建分步偏好数据的新方法，该方法利用了决策树中此前被忽略的失败探索。在后续的训练阶段，我们首先使用成功的工具使用专家轨迹对LLM进行微调，然后利用偏好数据应用直接偏好优化（DPO）来更新LLM的策略，由此得到我们的ToolPrefer-LLaMA（TP-LLaMA）模型。该方法不仅提升了原始专家数据的利用率，还拓宽了模型的学习空间。我们的实验表明，通过从推理树的错误中获取洞见，TP-LLaMA在几乎所有测试场景中都以大幅优势显著优于基线模型，并且在未见过的API上展现出更好的泛化能力。同时，TP-LLaMA还表现出优于基线模型的推理效率，使其更适合复杂的工具使用推理任务。
英文摘要：
Tool-augmented large language models (LLMs) leverage tools, often in the form of APIs, to improve their reasoning capabilities on complex tasks. This enables them to act as intelligent agents interacting with the real world. The recently introduced ToolLLaMA model by Qin et al. [2023] utilizes the depth-first search-based decision tree (DFSDT) mechanism for multi-step reasoning with $16000+$ real-world APIs, effectively enhancing the performance of tool-augmented LLMs compared to traditional chain reasoning mechanisms. However, their approach only employs successful paths from decision trees (also called inference trees) for supervised fine-tuning (SFT), missing out on the potential learning opportunities from failed paths. Inspired by this, we propose an inference trajectory optimization framework based on preference learning to address this limitation. We first introduce a novel method for constructing step-wise preference data from tree-like expert trajectories, which leverages the previously ignored failed explorations in the decision trees. In the subsequent training phase, we first fine-tune the LLM with successful tool-usage expert trajectories and then apply direct preference optimization (DPO) with the preference data to update the LLM's policy, resulting in our ToolPrefer-LLaMA (TP-LLaMA) model. This approach not only enhances the utilization of original expert data but also broadens the learning space of the model. Our experiments demonstrate that by obtaining insights from errors in inference trees, TP-LLaMA significantly outperforms the baselines across almost all test scenarios by a large margin and exhibits better generalization capabilities with unseen APIs. At the same time, TP-LLaMA has also demonstrated superior reasoning efficiency compared to the baselines, making it more suitable for complex tool-usage reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
