通过蒙特卡洛树搜索的过程监督增强推理能力
Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
AI总结：
本文提出利用蒙特卡洛树搜索让LLM自生成过程监督数据，通过加权对数似然迭代训练，在数学推理任务上显著提升性能并展现跨数据集迁移能力。
摘要：
大型语言模型（LLMs）已在多种任务中展现出卓越的能力。然而，推理对LLMs而言仍然是一项挑战。为了提高LLMs的推理能力，过程监督已被证明优于结果监督。在本工作中，我们研究使用蒙特卡洛树搜索（Monte Carlo Tree Search，MCTS）让LLMs自身生成过程监督数据以用于训练。我们利用LLM采样推理步骤，并为每一步分配一个反映其“相对正确性”的分数，然后通过最小化生成推理步骤的加权对数似然来训练LLM。这一“先生成后训练”的过程反复迭代直至收敛。我们的实验结果表明，所提出的方法在两个数学推理数据集上显著提升了LLMs的性能。此外，在一个数据集上训练的模型在另一个数据集上也表现出性能提升，显示出增强后的推理能力具有可迁移性。
英文摘要：
Large language models (LLMs) have demonstrated their remarkable capacity across a variety of tasks. However, reasoning remains a challenge for LLMs. To improve LLMs' reasoning ability, process supervision has proven to be better than outcome supervision. In this work, we study using Monte Carlo Tree Search (MCTS) to generate process supervision data with LLMs themselves for training them. We sample reasoning steps with an LLM and assign each step a score that captures its "relative correctness," and the LLM is then trained by minimizing weighted log-likelihood of generating the reasoning steps. This generate-then-train process is repeated iteratively until convergence.Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
