自训练激发大型语言模型的简洁推理
Self-Training Elicits Concise Reasoning in Large Language Models
AI总结：
本文提出利用 best-of-N 采样与少样本条件化生成简洁推理路径并进行自训练微调，在保持准确率的同时显著减少 LLM 的推理 token。
摘要：
思维链（Chain-of-thought，CoT）推理使大型语言模型（LLM）能够通过中间 token 利用额外计算来解决复杂任务。然而，我们认为典型的推理轨迹包含许多冗余 token，从而产生额外的推理成本。在检查当前 LLM 的输出分布时，我们发现有证据表明，相对于其默认行为，它们具备更简洁推理的潜在能力。为激发这一能力，我们提出了简单的微调方法，在任务特定设置中，利用通过 best-of-N 采样和少样本条件化获得的自生成简洁推理路径。我们的组合方法在 GSM8K 和 MATH 上跨五个模型系列平均减少 30% 的输出 token，同时保持平均准确率。通过利用 LLM 的基本随机性和上下文学习能力，我们的自训练方法能够在广泛模型上稳健地激发简洁推理，包括那些经过大量后训练的模型。代码可在 https://github.com/TergelMunkhbat/concise-reasoning 获取。
英文摘要：
Chain-of-thought (CoT) reasoning has enabled large language models (LLMs) to utilize additional computation through intermediate tokens to solve complex tasks. However, we posit that typical reasoning traces contain many redundant tokens, incurring extraneous inference costs. Upon examination of the output distribution of current LLMs, we find evidence on their latent ability to reason more concisely, relative to their default behavior. To elicit this capability, we propose simple fine-tuning methods which leverage self-generated concise reasoning paths obtained by best-of-N sampling and few-shot conditioning, in task-specific settings. Our combined method achieves a 30% reduction in output tokens on average, across five model families on GSM8K and MATH, while maintaining average accuracy. By exploiting the fundamental stochasticity and in-context learning capabilities of LLMs, our self-training approach robustly elicits concise reasoning on a wide range of models, including those with extensive post-training. Code is available at https://github.com/TergelMunkhbat/concise-reasoning
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
