面向高效思维链推理的价值引导搜索
Value-Guided Search for Efficient Chain-of-Thought Reasoning
AI总结：
针对长上下文推理中过程奖励建模难以定义步骤的问题，提出token级价值模型与分块VGS搜索，并用加权多数投票提升测试时扩展效率、降低推理FLOPs。
摘要：
本文提出一种简单高效的方法，用于在长上下文推理轨迹上训练价值模型。与现有过程奖励模型（PRM）不同，该方法不需要细粒度的“步骤”概念，而这种概念在长上下文推理模型中很难定义。通过收集包含250万条推理轨迹的数据集，我们训练了一个1.5B参数的token级价值模型，并将其应用于DeepSeek模型，以借助测试时计算扩展提升性能。我们发现，结合最终加权多数投票的分块价值引导搜索（VGS）相比多数投票或best-of-n等标准方法，实现了更好的测试时扩展。此外，在达到与多数投票相同性能时，VGS显著降低了所需的推理FLOPs。我们的数据集、模型和代码库均已开源。
英文摘要：
In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
