基于令牌级适应性潜在链式思维的预训练
Pretraining with Token-Level Adaptive Latent Chain-of-Thought
AI总结：
本研究提出了一种基于令牌级适应性潜在链式思维的预训练方法，通过内部化潜在链式思维来提高模型性能，减少计算成本，并在语言建模和下游任务中取得显著改进。
摘要：
通过增加参数和训练数据来扩展大语言模型正变得越来越受限于高质量语料库的限制和日益上升的通信成本。这项工作探索了一个替代方向：通过内部化潜在链式思维（CoT）来增加每令牌的计算量，而无需扩展参数。我们提出了基于令牌级适应性潜在链式思维的预训练（adaptive latent CoT），其中模型在生成每个令牌之前会生成一个可变长度的潜在CoT轨迹——为困难的令牌分配更长的轨迹，为简单的令牌分配较短（甚至零）的轨迹。重要的是，这种行为自然地从在通用文本上的一次性预训练中产生，并通过令牌级适应性终止来减少训练和推理中的计算量。实验表明，使用Llama架构时，适应性潜在CoT在语言建模困惑度和广泛下游准确性方面持续改进，即使训练FLOPs少于先前的递归基线。
英文摘要：
Scaling large language models by increasing parameters and training data is increasingly constrained by limited high-quality corpora and rising communication costs. This work explores an alternative axis: increasing per-token computation without expanding parameters, by internalizing latent Chain-of-Thought (CoT) into pretraining. We propose Pretraining with Token-Level Adaptive Latent CoT (adaptive latent CoT), where the model generates a variable-length latent CoT trajectory before emitting each token -- allocating longer trajectories to difficult tokens and shorter (or even zero) trajectories to easy ones. Importantly, this behavior emerges naturally from one-stage pretraining on general text and reduces computation in both training and inference via token-wise adaptive halting. Experiments with Llama architectures show that adaptive latent CoT consistently improves language modeling perplexity and broad downstream accuracy, even with fewer training FLOPs than prior recurrent baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
