学习排序思维链：使用小型模型
Learning to Rank Chain-of-Thought: Using a Small Model
AI总结：
针对大语言模型数学推理可靠性差、验证成本高的问题，提出仅55M参数的轻量能量结果奖励模型EORM，用结果标签训练以排序思维链解，提升小模型推理准确率且泛化性好，兼具效率与鲁棒性。
摘要：
大语言模型（LLMs）在可靠的数学推理方面存在困难，且当前的验证方法通常计算成本高昂。本文提出了能量结果奖励模型（Energy Outcome Reward Model，EORM），这是一种高效轻量的事后验证器，旨在应对这一挑战。EORM采用基于能量的框架对思维链（Chain-of-Thought，CoT）解决方案进行排序，仅利用简单的结果标签学习区分正确与错误推理，从而无需昂贵的标注工作。EORM仅含5500万参数，比典型奖励模型小127倍以上，它将Llama 3 8B在GSM8k上的准确率提升至90.7%，在MATH上的准确率提升至63.7%。该性能通过从候选池中高效选择最优推理路径实现，使其能够匹配甚至超越资源消耗高得多的Best-of-N采样技术的准确率。至关重要的是，我们的实验表明EORM能有效泛化到分布外问题和未见过的模型，这表明它学习到了有效推理的基本原理。这种鲁棒性与其高效性相结合，使EORM成为在复杂现实应用中部署更可靠LLMs的实用工具。
英文摘要：
Large Language Models (LLMs) struggle with reliable mathematical reasoning, and current verification methods are often computationally expensive. This paper introduces the Energy Outcome Reward Model (EORM), a highly efficient, lightweight post-hoc verifier designed to address this challenge. EORM uses an energy-based framework to rank Chain-of-Thought (CoT) solutions, learning to distinguish correct from incorrect reasoning using only simple outcome labels, thus eliminating the need for expensive annotations. With only 55M parameters, over 127 times smaller than typical reward models, EORM boosts the accuracy of Llama 3 8B to 90.7\% on GSM8k and 63.7\% on MATH. This performance is achieved by efficiently selecting the optimal reasoning path from a pool of candidates, allowing it to match or exceed the accuracy of far more resource-intensive Best-of-N sampling techniques. Crucially, our experiments show that EORM generalizes effectively to out-of-distribution problems and unseen models, indicating it learns fundamental principles of valid reasoning. This robustness, combined with its efficiency, establishes EORM as a practical tool for deploying more dependable LLMs in complex, real-world applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
