训练语言模型进行高效推理
Training Language Models to Reason Efficiently
AI总结：
本文提出使用强化学习训练大型推理模型，根据任务复杂度动态分配推理时计算资源，在保持准确率的同时显著降低推理成本，实现高效推理。
摘要：
扩大模型规模和训练数据已为大型语言模型（LLMs）的性能带来了巨大提升。然而，这种方法的边际效益递减，需要替代方法来提升模型能力，尤其是在需要高级推理的任务中。利用长思维链的大型推理模型在解决问题能力上带来了前所未有的突破，但代价是与更长生成相关的巨大部署成本。降低推理成本对于这些模型的经济可行性、用户体验和环境可持续性至关重要。在这项工作中，我们提出训练大型推理模型以进行高效推理。更准确地说，我们使用强化学习（RL）训练推理模型，根据任务复杂度动态分配推理时计算资源。我们的方法促使模型最小化不必要的计算开销，同时保持准确率，从而实现显著的效率提升。它能够推导出一系列具有不同效率水平的推理模型，这些模型通过单一超参数控制。在两个开放权重的大型推理模型上的实验表明，在保留大部分准确率的同时，推理成本显著降低。
英文摘要：
Scaling model size and training data has led to great advances in the performance of Large Language Models (LLMs). However, the diminishing returns of this approach necessitate alternative methods to improve model capabilities, particularly in tasks requiring advanced reasoning. Large reasoning models, which leverage long chain-of-thoughts, bring unprecedented breakthroughs in problem-solving capabilities but at a substantial deployment cost associated to longer generations. Reducing inference costs is crucial for the economic feasibility, user experience, and environmental sustainability of these models. In this work, we propose to train large reasoning models to reason efficiently. More precisely, we use reinforcement learning (RL) to train reasoning models to dynamically allocate inference-time compute based on task complexity. Our method incentivizes models to minimize unnecessary computational overhead while maintaining accuracy, thereby achieving substantial efficiency gains. It enables the derivation of a family of reasoning models with varying efficiency levels, controlled via a single hyperparameter. Experiments on two open-weight large reasoning models demonstrate significant reductions in inference cost while preserving most of the accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
