思维链推理是一种策略改进算子
Chain-of-Thought Reasoning is a Policy Improvement Operator
AI总结：
针对大语言模型依赖人类训练数据、缺乏自学能力的问题，提出SECToR框架，通过思维链推理构建自学习循环，验证了思维链可作为策略改进算子，助力模型在有限初始数据下自主习得更复杂的技能。
摘要：
大语言模型凭借令人惊叹的全新能力震惊了世界，但目前它们尚不具备自学新技能的能力，反而依赖大量人类生成的训练数据。我们提出SECToR（通过思维链推理实现自我教育），这一概念验证表明语言模型能够利用思维链推理自学新技能。在自学习循环中，SECToR要求模型使用思维链推理解答加法问题，随后训练下一版本的模型在不使用此类推理的情况下直接解答这些相同的问题。这一过程通常会得到性能提升的模型，当再次辅以思维链推理时，该模型能够解答比原始模型更难的问题，从而让自学习循环得以持续。通过SECToR训练的语言模型，在仅包含6位及以下数字的初始监督微调阶段之外，无需接触任何真实标注样本，就能自主学会解答最长位数数字的加法。我们的核心假设是，思维链推理可充当策略改进算子，其作用类似于AlphaZero（Silver等人，2017）中使用的蒙特卡洛树搜索。我们希望这项研究能开辟新方向，让语言模型无需人类示范就能学会自我教学。
英文摘要：
Large language models have astounded the world with fascinating new capabilities. However, they currently lack the ability to teach themselves new skills, relying instead on large amounts of human-generated training data. We introduce SECToR (Self-Education via Chain-of-Thought Reasoning), a proof-of-concept demonstration that language models can teach themselves new skills using chain-of-thought reasoning. During the self-learning loop, SECToR asks models to solve addition problems using chain-of-thought reasoning before training the next version of the model to solve those same problems directly without using such reasoning. This process often results in an improved model which can, when again augmented with chain-of-thought reasoning, solve even harder problems than the original model, allowing the self-learning loop to continue. Language models trained via SECToR autonomously learn to add up to the longest-length-digit numbers without access to any ground truth examples beyond an initial supervised fine-tuning phase consisting only of numbers with 6 or fewer digits. Our central hypothesis is that chain-of-thought reasoning can act as a policy improvement operator, similarly to how Monte-Carlo Tree Search is used in AlphaZero (Silver et al., 2017). We hope that this research can lead to new directions in which language models can learn to teach themselves without the need for human demonstrations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
