发表机构
Google; School of Computing, Dublin City University（谷歌公司; 都柏林城市大学计算机学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出训练混合（MoT）框架，将Transformer划分为层块在冻结对齐器内独立训练后重组，在13亿参数Gemma模型上验证其可重组为可用语言模型，可复用对齐器实现计算优势，用于研究可复用训练单元。
AI 中文摘要
我们探究语言模型预训练是否可分解为更小的、可独立训练的任务，后续能重组为一个连贯的更大模型。我们提出训练混合（Mixture of Training, MoT），这是一种支架式模块化预训练流程，它将目标Transformer划分为连续的层块，在冻结的预训练对齐器支架内训练每个块，之后重组这些已训练块，还可选择进行一次简短的端到端适配过程。在基于C4训练的13亿参数Gemma风格模型上，MoT提供了小规模的机制验证：独立训练的深度切片可重组为可用的语言模型，且质量对等方案达到了与整体基线相同的困惑度。该对等设置在对齐器准备后，处理了更多的聚合令牌，且具有更短的理想化层等效关键路径；其有效计算优势取决于跨运行复用对齐器。因此，我们提出MoT并非整体预训练的通用替代品，而是用于研究支架式子运行能否作为可复用训练单元的小规模框架。
英文摘要
We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass. On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slices can be recomposed into a usable language model, and a quality-parity schedule reaches the same reported perplexity as the monolithic baseline. This parity setting processes more aggregate tokens and has a shorter idealized layer-equivalent critical path after aligner preparation; its effective compute advantage depends on reusing the aligner across runs. We therefore present MoT not as a general replacement for monolithic pre-training, but as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.
CommentsAccepted at the Workshop on Methods and Opportunities at Small Scale (MOSS), COLM 2026
