arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.07129cs.LGcs.CL
收藏

生长变换器:在固定基质上的模块化组合与逐层扩展

Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate

Moscow Institute of Physics and Technology (MIPT)(莫斯科物理技术学院)

A. Bochkov

AI总结:

本文研究了在固定接口下训练解码器Transformer的约束训练方法,通过逐层扩展模型并保持可训练参数预算,展示了在有限参数下持续学习的可行性及与密集训练的权衡。

摘要:

我们研究了一种约束训练模式,用于解码器-only的Transformer,其中令牌接口固定,先前训练的密集块不被重新打开,且随着深度增长,可训练参数集保持大致恒定。从浅层模型开始,我们堆叠新块并只训练最新块和语言模型头部;可选的LoRA阶段在相同可训练参数预算下提供有限的全局调整。本文探讨了可行性与权衡问题,而非是否这种模式匹配于微调的单体预训练。在一项常见的协议9层研究中,冻结的Unicode基质上构造的冻结-Unicode模型使用了105.0M个可训练参数,与接口匹配的单体冻结基线的180.5M和完全可训练的单体基线的247.6M相比。我们随后考虑了一个极端的固定接口:每个令牌仅由冻结的16维二进制令牌ID代码表示,确定性地提升到d_model,因此生成的令牌嵌入矩阵的秩最多为16。即使在这种设定下,持续增长仍然可行。在FineWeb-Edu + Cosmopedia上进行的68.9B-token运行中,一个16层、269.7M参数的模型在经过LoRA阶段后,达到了28.92%的MMLU成绩。报告的最终指标是在将最后阶段的LoRA适配器合并到269.7M基础模型后测量的。由于在此长期运行中数据混合在不同阶段发生变化,我们将其解释为可行性演示,而非干净的因果比较。总体而言,证据支持一个狭窄的结论:在有限的可训练参数预算下,可以在冻结的最小接口之上进行有用的持续学习,但与密集单体训练相比,最终困惑度存在明显权衡。

英文摘要:

We study a constrained training regime for decoder-only Transformers in which the token interface is fixed, previously trained dense blocks are not reopened, and the active trainable parameter set is kept approximately constant as depth grows. Starting from a shallow model, we stack new blocks and train only the newest blocks and the LM head; optional LoRA phases provide limited global readjustment under the same active-parameter budget. The paper asks a feasibility/tradeoff question, not whether this regime matches tuned monolithic pretraining. In a common-protocol 9-layer study on a frozen Unicode substrate, the constructive frozen-Unicode model uses 105.0M active trainable parameters, compared with 180.5M for the interface-matched monolithic frozen baseline and 247.6M for the fully trainable monolithic baseline. We then consider an extreme fixed interface: each token is represented only by a frozen 16-dim binary token-ID code, deterministically lifted to d_model, so the resulting token embedding matrix has rank at most 16. Even in this setting, continued growth remains viable. In a 68.9B-token run on FineWeb-Edu + Cosmopedia, a 16-layer 269.7M model trained above this fixed interface reaches 28.92\% MMLU after an interleaved LoRA stage. Reported final metrics are measured after merging the last-stage LoRA adapters into the 269.7M base model. Because the data mixture changes across stages in this long-horizon run, we interpret it as a viability demonstration rather than a clean causal comparison. Overall, the evidence supports a narrow claim: useful continued learning can proceed above a frozen minimal interface under a bounded active trainable-parameter budget, with a clear tradeoff against dense monolithic training in final perplexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑