智能扩展：通过小模型初始化加速大型语言模型预训练
Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
AI总结：
本文提出HyperCloning方法，通过使用预训练的小模型参数初始化大模型，在保留小模型功能与精度的同时显著减少大型语言模型预训练所需的GPU时间。
摘要：
语言模型的预训练阶段通常从随机初始化的参数开始。随着当前模型扩展的趋势，训练其大量参数可能极其缓慢且成本高昂。相比之下，小型语言模型的训练成本较低，但它们往往无法达到大型模型的准确度。在本文中，我们探索了一个连接这两种不同情况的有趣想法：我们能否开发一种方法，使用较小的预训练模型来初始化大型语言模型？这种初始化是否会在训练时间和最终准确度方面带来任何好处？在本文中，我们引入了HyperCloning，这是一种能够将预训练语言模型的参数扩展为具有更大隐藏维度的更大模型参数的方法。我们的方法确保更大的模型保留较小模型的功能。因此，在训练开始之前，更大的模型已经继承了较小模型的预测能力和准确度。我们证明，训练这样一个已初始化的模型，可以显著节省预训练大型语言模型所需的GPU小时数。
英文摘要：
The pre-training phase of language models often begins with randomly initialized parameters. With the current trends in scaling models, training their large number of parameters can be extremely slow and costly. In contrast, small language models are less expensive to train, but they often cannot achieve the accuracy of large models. In this paper, we explore an intriguing idea to connect these two different regimes: Can we develop a method to initialize large language models using smaller pre-trained models? Will such initialization bring any benefits in terms of training time and final accuracy? In this paper, we introduce HyperCloning, a method that can expand the parameters of a pre-trained language model to those of a larger model with increased hidden dimensions. Our method ensures that the larger model retains the functionality of the smaller model. As a result, the larger model already inherits the predictive power and accuracy of the smaller model before the training starts. We demonstrate that training such an initialized model results in significant savings in terms of GPU hours required for pre-training large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
