通过自动化过程监督提升语言模型的数学推理能力
Improve Mathematical Reasoning in Language Models by Automated Process Supervision
AI总结：
针对大语言模型多步推理中过程监督数据收集成本高的问题，提出OmegaPRM分治式MCTS算法自动收集标注，结合加权自一致性显著提升了多个模型的数学推理成功率。
摘要：
复杂的多步推理任务，如求解数学问题或生成代码，即便对最先进的大语言模型（LLM）而言仍是一项重大挑战。使用结果奖励模型（Outcome Reward Model, ORM）验证LLM输出是推理阶段提升LLM推理性能的标准技术。然而，对于推理链较长或涉及多跳的推理任务，这种方法仍显不足，因为中间结果既无法得到恰当的奖励，也不会受到相应的惩罚。过程监督通过在推理过程中分配中间奖励来解决这一局限。迄今为止，用于收集过程监督数据的方法要么依赖人工标注，要么依赖每步蒙特卡洛估计，二者的扩展成本都极高，从而阻碍了该技术的广泛应用。针对这一挑战，我们提出了一种名为OmegaPRM的新型分治式蒙特卡洛树搜索（Monte Carlo Tree Search, MCTS）算法，用于高效收集高质量的过程监督数据。该算法通过二分搜索快速定位思维链（Chain of Thought, CoT）中的首个错误，并平衡正负样本，从而兼顾效率与质量。最终，我们得以收集超过150万条过程监督标注，用于训练过程奖励模型（Process Reward Models, PRMs）。这种完全自动化的过程监督结合加权自一致性算法，能够提升LLM的数学推理性能。我们将经过指令微调的Gemini Pro模型在MATH500上的成功率从51%提升至69.4%，在GSM8K上从86.4%提升至93.6%。同样，我们将Gemma2 27B在MATH500上的成功率从42.3%提升至58.2%，在GSM8K上从74.0%提升至92.2%。整个过程无需任何人工干预或监督，使得我们的方法在成本和……
英文摘要：
Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a lengthy or multi-hop reasoning chain, where the intermediate outcomes are neither properly rewarded nor penalized. Process supervision addresses this limitation by assigning intermediate rewards during the reasoning process. To date, the methods used to collect process supervision data have relied on either human annotation or per-step Monte Carlo estimation, both prohibitively expensive to scale, thus hindering the broad application of this technique. In response to this challenge, we propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named \textit{OmegaPRM} for the efficient collection of high-quality process supervision data. This algorithm swiftly identifies the first error in the Chain of Thought (CoT) with binary search and balances the positive and negative examples, thereby ensuring both efficiency and quality. As a result, we are able to collect over 1.5 million process supervision annotations to train Process Reward Models (PRMs). This fully automated process supervision alongside the weighted self-consistency algorithm is able to enhance LLMs' math reasoning performances. We improved the success rates of the instruction-tuned Gemini Pro model from 51\% to 69.4\% on MATH500 and from 86.4\% to 93.6\% on GSM8K. Similarly, we boosted the success rates of Gemma2 27B from 42.3\% to 58.2\% on MATH500 and from 74.0\% to 92.2\% on GSM8K. The entire process operates without any human intervention or supervision, making our method both financially and ...
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
