一种变换器架构的修改以激励外部化推理
A transformer architecture alteration to incentivise externalised reasoning
AI总结:
本文提出一种变换器架构修改和训练流程,使大语言模型更倾向于产生详细推理。通过在中间层加入早退机制,训练模型在预测下一个token时提前退出计算,从而减少冗余计算。
摘要:
我们提出了一种新的架构变更和训练流程,使LLMs成为更详尽的推理者。我们通过在现有变换器架构中加入中间层的早退机制,并训练模型在预测下一个token时无需深入计算即可在更浅的层退出。经过校准阶段后,我们利用强化学习激励模型尽可能早地退出,同时保持任务性能。我们为小型推理模型提供了初步结果,显示它们能够自适应地减少跨token的计算量。我们预测,当应用于合适规模时,我们的方法可以最小化推理模型在内部激活中用于非远视规划的冗余计算量,仅将此保留给难以预测的token。
英文摘要:
We propose a new architectural change, and post-training pipeline, for making LLMs more verbose reasoners by teaching a model to truncate forward passes early. We augment an existing transformer architecture with an early-exit mechanism at intermediate layers and train the model to exit at shallower layers when the next token can be predicted without deep computation. After a calibration stage, we incentivise the model to exit as early as possible while maintaining task performance using reinforcement learning. We provide preliminary results to this effect for small reasoning models, showing that they learn to adaptively reduce computations across tokens. We predict that, applied at the right scale, our approach can minimise the amount of excess computation that reasoning models have at their disposal to perform non-myopic planning using their internal activations, reserving this only for difficult-to-predict tokens.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。