让我们逐步奖励:逐步奖励模型作为推理的导航者
Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
AI总结:
本文提出利用过程监督奖励模型的逐步反馈,通过启发式贪心搜索优化大语言模型的多步推理路径,在数学和代码生成任务上均取得优于思维链的效果。
摘要:
近年来,大型语言模型(LLMs)在多步推理方面取得了显著进展。先前的研究已经阐明了在模型推理过程中整合反馈或搜索机制以提高推理准确性的优点。过程监督奖励模型(PRM)通常在训练阶段为LLMs提供逐步反馈,类似于近端策略优化(PPO)或拒绝采样。我们的目标是检验PRM在推理阶段的有效性,以帮助识别多步任务(如数学推理和代码生成)的最优解路径。为此,我们提出了一种启发式贪心搜索算法,利用PRM的逐步反馈来优化LLMs探索的推理路径。这种定制化的PRM在GSM8K和MATH等数学基准上相比思维链(CoT)展示了更好的结果。此外,为了探索我们方法的通用性,我们开发了一种新方法来自动生成编码任务的逐步奖励数据集,并在代码生成任务中观察到类似的性能提升。这突显了我们基于奖励模型的推理方法在推理任务中的鲁棒性。
英文摘要:
Recent years have seen considerable advancements in multi-step reasoning with Large Language Models (LLMs). The previous studies have elucidated the merits of integrating feedback or search mechanisms during model inference to improve the reasoning accuracy. The Process-Supervised Reward Model (PRM), typically furnishes LLMs with step-by-step feedback during the training phase, akin to Proximal Policy Optimization (PPO) or reject sampling. Our objective is to examine the efficacy of PRM in the inference phase to help discern the optimal solution paths for multi-step tasks such as mathematical reasoning and code generation. To this end, we propose a heuristic greedy search algorithm that employs the step-level feedback from PRM to optimize the reasoning pathways explored by LLMs. This tailored PRM demonstrated enhanced results compared to the Chain of Thought (CoT) on mathematical benchmarks like GSM8K and MATH. Additionally, to explore the versatility of our approach, we develop a novel method to automatically generate step-level reward dataset for coding tasks and observed similar improved performance in the code generation tasks. Thus highlighting the robust nature of our reward-model-based approach to inference for reasoning tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。