通过协同验证扩展推理计算以提升大语言模型推理能力
Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification
AI总结：
针对大语言模型推理准确性不足的问题，研究通过构建正误解答数据集训练验证器，结合CoT与PoT的协同验证方法扩展推理计算，在数学、代码推理基准上取得SOTA结果。
摘要：
尽管大语言模型（LLM）的通用能力已取得显著进展，但它们仍难以实现稳定准确的推理，在数学推理、代码推理等复杂任务中尤为突出。一个关键局限在于，LLM主要基于正确解答进行训练，这削弱了其检测错误并从错误中学习的能力，进而阻碍了其对输出结果进行可靠验证与排序的能力。为解决这一问题，我们通过生成多条推理路径，并利用验证器根据正确性对生成的输出进行评估与排序，扩大了推理阶段的计算量。为支撑这一方案，我们构建了一个包含数学与代码任务正确及错误解答的综合数据集，这些解答由多个LLM生成。这种多样化的解答集合使验证器能够更有效地区分正确答案与错误输出并完成排序。我们通过对现有方法的广泛对比，选定了构建验证器的训练方法。此外，为发挥不同推理策略的独特优势，我们提出了一种新颖的协同方法，整合思维链（Chain-of-Thought, CoT）与思维程序（Program-of-Thought, PoT）解答用于验证。CoT提供了清晰的逐步推理过程，提升了可解释性；而PoT具备可执行性，提供了精确且对错误敏感的验证机制。结合二者的优势，我们的方法显著提升了推理验证的准确性与可靠性。我们的验证器Math-Rev和Code-Rev相较现有LLM展现出大幅性能提升，在GSM8k、MATH等基准测试上取得了当前最优结果，在以Qwen-72B-Instruct作为推理器时甚至优于GPT-4o。
英文摘要：
Despite significant advancements in the general capability of large language models (LLMs), they continue to struggle with consistent and accurate reasoning, especially in complex tasks such as mathematical and code reasoning. One key limitation is that LLMs are trained primarily on correct solutions, reducing their ability to detect and learn from errors, which hampers their ability to reliably verify and rank outputs. To address this, we scale up the inference-time computation by generating multiple reasoning paths and employing verifiers to assess and rank the generated outputs by correctness. To facilitate this, we introduce a comprehensive dataset consisting of correct and incorrect solutions for math and code tasks, generated by multiple LLMs. This diverse set of solutions enables verifiers to more effectively distinguish and rank correct answers from erroneous outputs. The training methods for building verifiers were selected based on an extensive comparison of existing approaches. Moreover, to leverage the unique strengths of different reasoning strategies, we propose a novel collaborative method integrating Chain-of-Thought (CoT) and Program-of-Thought (PoT) solutions for verification. CoT provides a clear, step-by-step reasoning process that enhances interpretability, while PoT, being executable, offers a precise and error-sensitive validation mechanism. By taking both of their strengths, our approach significantly improves the accuracy and reliability of reasoning verification. Our verifiers, Math-Rev and Code-Rev, demonstrate substantial performance gains to existing LLMs, achieving state-of-the-art results on benchmarks such as GSM8k and MATH and even outperforming GPT-4o with Qwen-72B-Instruct as the reasoner.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
