超越沉没成本:通过专家混合的正交增长提升LLM预训练效率
Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
AI总结:
本文提出正交增长策略,通过优化专家混合模型的深度和宽度,提升LLM预训练效率,实验证明在相同计算预算下,模型准确率提升10.6%。
摘要:
随着预训练大型语言模型的计算需求持续增加,高效训练范式变得至关重要。尽管现有预训练检查点已投入大量资源,但受限于架构限制,这些资产往往未被充分利用。我们引入一种
英文摘要:
As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical. Despite the vast resources already invested in existing pre-trained checkpoints, these assets often remain under-leveraged due to architectural limitations. We introduce an "orthogonal growth" strategy designed to "recycle" these checkpoints by strategically expanding their parameters prior to continued training. Our method focuses on optimizing converged Mixture-of-Experts (MoE) models through two dimensions: interpositional layer copying for increased depth and noisy expert duplication for expanded width. Through extensive scaling laws analysis, we demonstrate a strong positive correlation between the "sunk cost" (prior investment) and the final model accuracy. Empirical results on models up to 70B parameters and 1T tokens show that our recycling approach yields a 10.6% accuracy improvement compared to training from scratch under identical extra compute budgets. This work provides a cost-effective blueprint for sustainable large-scale LLM development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。