思考型LLM：基于思维生成的通用指令遵循
Thinking LLMs: General Instruction Following with Thought Generation
AI总结：
提出一种无需额外人类数据的迭代搜索与偏好优化训练方法，使LLM在回答前具备显式思考能力，在AlpacaEval和Arena-Hard等基准及非推理任务上均取得显著提升。
摘要：
LLM通常被训练成像人类专家一样回答用户问题或遵循指令。然而，在标准对齐框架中，它们缺乏在回答前进行显式思考的基本能力。思考对于需要推理和规划的复杂问题非常重要——但也可应用于任何任务。我们提出一种训练方法，无需使用额外的人类数据，即可为现有LLM配备这种思考能力以实现通用指令遵循。我们通过迭代搜索和优化过程来实现这一点，该过程探索可能的思维生成空间，使模型能够在没有直接监督的情况下学习如何思考。对于每个指令，使用评判模型对思维候选进行评分，仅评估其响应，然后通过偏好优化进行优化。我们证明，此过程在AlpacaEval和Arena-Hard上带来了卓越的性能，并且在营销、健康和常识等非推理类别，以及更传统的推理与问题解决任务中，都显示出思考带来的收益。
英文摘要：
LLMs are typically trained to answer user questions or follow instructions similarly to how human experts respond. However, in the standard alignment framework they lack the basic ability of explicit thinking before answering. Thinking is important for complex questions that require reasoning and planning -- but can be applied to any task. We propose a training method for equipping existing LLMs with such thinking abilities for general instruction following without use of additional human data. We achieve this by an iterative search and optimization procedure that explores the space of possible thought generations, allowing the model to learn how to think without direct supervision. For each instruction, the thought candidates are scored using a judge model to evaluate their responses only, and then optimized via preference optimization. We show that this procedure leads to superior performance on AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning categories such as marketing, health and general knowledge, in addition to more traditional reasoning & problem-solving tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
