通过符号引导的蒙特卡洛过程监督增强语言模型的逻辑推理能力
Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
AI总结：
针对LLMs因记忆而非泛化导致逻辑推理鲁棒性不足的问题，提出基于蒙特卡洛估计合成带伪标签的符号推理轨迹，结合PRM、DPO和SFT进行训练，有效提升了模型的逻辑推理与领域外泛化能力。
摘要：
大型语言模型（LLMs）在许多推理基准测试中表现出色。然而，最近的研究指出，记忆而非泛化是导致这种性能的主要原因之一。事实上，LLMs容易受到内容变化的影响，表明其缺乏支持推理过程的稳健规划或符号抽象。为了提高可靠性，人们已经做出了许多将LLMs与符号方法相结合的尝试。然而，由于开发可靠且可扩展的验证机制存在挑战，现有方法未能有效利用符号表示。在本文中，我们提出通过蒙特卡洛估计大规模合成带有逐步伪标签的高质量符号推理轨迹，以克服这些局限性。基于合成数据可以高效训练过程奖励模型（PRM），然后利用它选择更多的符号轨迹。这些轨迹随后与直接偏好优化（DPO）和监督微调（SFT）相结合，以改善逻辑推理和泛化能力。我们在基准测试（即FOLIO和LogicAsker）上的结果表明了所提方法的有效性，并在前沿模型和开放权重模型上取得了增益。此外，在声明验证数据上的额外实验表明，在生成的符号推理轨迹上进行微调可增强领域外的泛化能力，这表明所提方法在增强规划和逻辑推理方面具有潜在影响。
英文摘要：
Large language models (LLMs) have shown strong performance in many reasoning benchmarks. However, recent studies have pointed to memorization, rather than generalization, as one of the leading causes for such performance. LLMs, in fact, are susceptible to content variations, demonstrating a lack of robust planning or symbolic abstractions supporting their reasoning process. To improve reliability, many attempts have been made to combine LLMs with symbolic methods. Nevertheless, existing approaches fail to effectively leverage symbolic representations due to the challenges involved in developing reliable and scalable verification mechanisms. In this paper, we propose to overcome such limitations by synthesizing high-quality symbolic reasoning trajectories with stepwise pseudo-labels at scale via Monte Carlo estimation. A Process Reward Model (PRM) can be efficiently trained based on the synthesized data and then used to select more symbolic trajectories. The trajectories are then employed with Direct Preference Optimization (DPO) and Supervised Fine-Tuning (SFT) to improve logical reasoning and generalization. Our results on benchmarks (i.e., FOLIO and LogicAsker) show the effectiveness of the proposed method with gains on frontier and open-weight models. Moreover, additional experiments on claim verification data reveal that fine-tuning on the generated symbolic reasoning trajectories enhances out-of-domain generalizability, suggesting the potential impact of the proposed method in enhancing planning and logical reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
