通过轨迹收集与过程奖励合成学习基于规划的推理
Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing
AI总结：
针对LLM推理中的幻觉、规划搜索高延迟及人工标注成本高的问题，提出通过收集轨迹并基于合成过程奖励进行DPO来学习基于规划的推理，其7B模型在逻辑推理基准上超越了GPT-3.5-Turbo。
摘要：
大型语言模型（LLMs）通过逐步生成理据，在处理复杂推理任务方面展现了巨大潜力。然而，近期研究对其推理过程中的幻觉和缺陷提出了担忧。为提升生成理据的可靠性与忠实度，人们付出了大量努力。一些方法将推理建模为规划，另一些则专注于为过程监督进行标注。然而，由于频繁评估中间推理状态和庞大的探索空间，基于规划的搜索过程常导致高延迟。此外，用人工标注监督推理过程成本高昂，且难以扩展以用于 LLM 训练。为解决这些问题，本文提出一个框架，通过对收集的轨迹进行 Direct Preference Optimization (DPO) 来学习基于规划的推理，这些轨迹根据合成的过程奖励进行排序。我们在具有挑战性的逻辑推理基准上的结果证明了该学习框架的有效性，表明我们的 7B 模型能够超越 GPT-3.5-Turbo 等强大的同类模型。
英文摘要：
Large Language Models (LLMs) have demonstrated significant potential in handling complex reasoning tasks through step-by-step rationale generation. However, recent studies have raised concerns regarding the hallucination and flaws in their reasoning process. Substantial efforts are being made to improve the reliability and faithfulness of the generated rationales. Some approaches model reasoning as planning, while others focus on annotating for process supervision. Nevertheless, the planning-based search process often results in high latency due to the frequent assessment of intermediate reasoning states and the extensive exploration space. Additionally, supervising the reasoning process with human annotation is costly and challenging to scale for LLM training. To address these issues, in this paper, we propose a framework to learn planning-based reasoning through Direct Preference Optimization (DPO) on collected trajectories, which are ranked according to synthesized process rewards. Our results on challenging logical reasoning benchmarks demonstrate the effectiveness of our learning framework, showing that our 7B model can surpass the strong counterparts like GPT-3.5-Turbo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
